Search

CN-122019493-A - Data management method, apparatus, device, medium, and program product

CN122019493ACN 122019493 ACN122019493 ACN 122019493ACN-122019493-A

Abstract

The application provides a data management method which can be applied to the technical field of big data. The data management method is executed in a user space file system and comprises the steps of receiving an operation request aiming at a first logic path file through a virtual file system, enabling the operation request to come from a user program, routing the operation request to a kernel driver, calling a callback function corresponding to the operation request, registering the callback function by the user program, inquiring a pre-established metadata base based on the callback function, obtaining metadata corresponding to the first logic path, wherein the metadata at least comprises a mapping relation between the logic path and a physical path, and executing operation corresponding to the operation request based on the metadata. The application also provides a data management device, equipment, a storage medium and a program product.

Inventors

  • HAN JUN

Assignees

  • 中国科学院国家天文台

Dates

Publication Date
20260512
Application Date
20260123

Claims (10)

  1. 1. A method of data management, the method for a user space file system comprising: receiving, by a virtual file system, an operation request for a first logical path file, the operation request from a user program; Routing the operation request to a kernel driver, and calling a callback function corresponding to the operation request, wherein the callback function is registered by the user program; inquiring a pre-established metadata base based on the callback function, and acquiring metadata corresponding to the first logic path, wherein the metadata at least comprises a mapping relation between the logic path and a physical path; and executing an operation corresponding to the operation request based on the metadata.
  2. 2. The method of claim 1, wherein the invoking the callback function corresponding to the operation request comprises: Encapsulating the operation request into a protocol message conforming to a user space file system framework through the kernel driver; Writing the protocol message into character equipment, wherein the character equipment is used for the communication between the kernel and the user space; reading and decoding the protocol message by monitoring a user space daemon of the character device; And calling a callback function corresponding to the operation request according to the decoding result of the protocol message.
  3. 3. The method of claim 2, wherein the operation request is a directory enumeration request and the callback function is a directory read function, the method comprising: calling the catalog reading function, inquiring the metadata base, and obtaining metadata with a logic father path as a first logic path; and extracting the file name of the metadata, and sending the file name to a user program through a virtual file system to form a directory list.
  4. 4. The method of claim 2, wherein the operation request is a file read request, the method comprising: Invoking a file opening function, querying the metadata base to determine whether a first physical path corresponding to the first logical path exists; If the first physical path exists, calling a file reading function in response to receiving the file reading request; Acquiring file data corresponding to the first physical path through the file reading function, wherein the file data is positioned in a storage server; and sending the file data to a user program through the virtual file system.
  5. 5. A method according to claim 3, characterized in that the method comprises: Responding to a received file screening request, inquiring the metadata base according to preset screening conditions, and acquiring a plurality of pieces of matching metadata, wherein the plurality of pieces of matching metadata have different levels of physical paths; modifying the logical parent paths of the plurality of matching metadata to the same logical path; and extracting file names of the modified pieces of matching metadata, and sending the file names to a user program through a virtual file system to form a screening directory list.
  6. 6. The method according to claim 1, characterized in that the method comprises: In response to receiving an alias creation instruction for a first logical path, creating at least one piece of new metadata in the metadata repository, the first logical path corresponding to initial metadata, a physical path of the initial metadata being a first physical path; Wherein the physical path of the at least one new metadata is the first physical path, and the logical path of the at least one new metadata is a second logical path, the second logical path being different from the first logical path; The first physical path is configured to correspond to the initial metadata and the at least one new piece of metadata.
  7. 7. A data management apparatus, the apparatus being based on a user space file system, comprising: A request receiving module for receiving a file operation request for a first logical path through a virtual file system, the operation request coming from a user program; the function calling module is used for routing the operation request to a kernel driver, calling a callback function corresponding to the operation request, and registering the callback function by the user program; The information acquisition module is used for inquiring a pre-established metadata base based on the callback function to acquire metadata corresponding to the first logic path, wherein the metadata at least comprises a mapping relation between the logic path and a physical path, and And the operation execution module is used for executing the operation corresponding to the operation request based on the metadata.
  8. 8. An electronic device, comprising: one or more processors; A memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-6.
  9. 9. A computer readable storage medium having stored thereon a computer program or instructions, which when executed by a processor, implement the steps of the method according to any of claims 1 to 6.
  10. 10. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Description

Data management method, apparatus, device, medium, and program product Technical Field The present application relates to the field of big data technologies, and in particular, to a data management method, apparatus, device, medium, and program product. Background With the rapid development of cloud computing, big data, artificial intelligence and other technologies, the data volume in various industry fields is rapidly increased. Data is typically stored centrally in distributed object storage or network attached storage systems in the form of mass files, which are handled by users by accessing computing resources deployed at a data center. In the related technology of data management for user access, the problems of low access efficiency, data redundancy, high storage cost and difficult maintenance of data consistency exist, and in addition, the logical view of the data cannot be flexibly organized according to the actual requirements of the user. Therefore, on the premise of ensuring the physical storage stability of data, it is of great importance to provide flexible customization and access to efficient logical data views for different users, application programs or task scenes, and simultaneously avoid the storage redundancy of the original space. Disclosure of Invention In view of the foregoing, embodiments of the present application provide a data management method, apparatus, device, medium, and program product. According to a first aspect of the application, a data management method is provided, and the method is used for a user space file system, and comprises the steps of receiving an operation request aiming at a first logic path file through a virtual file system, routing the operation request to a kernel driver, calling a callback function corresponding to the operation request, registering the callback function by the user program, inquiring a pre-established metadata base based on the callback function, acquiring metadata corresponding to the first logic path, wherein the metadata at least comprises a mapping relation between the logic path and a physical path, and executing operation corresponding to the operation request based on the metadata. According to the embodiment of the application, calling the callback function corresponding to the operation request comprises the steps of packaging the operation request into a protocol message conforming to a user space file system framework through a kernel driver, writing the protocol message into character equipment, wherein the character equipment is used for communicating with a user space through the kernel, reading and decoding the protocol message through a user space daemon of monitoring the character equipment, and calling the callback function corresponding to the operation request according to a decoding result of the protocol message. According to the embodiment of the application, the operation request is a directory enumeration request, the callback function is a directory reading function, and the method comprises the steps of calling the directory reading function, inquiring a metadata base, obtaining metadata with a logical parent path being a first logical path, extracting the file name of the metadata, and sending the file name to a user program through a virtual file system to form a directory list. According to the embodiment of the application, the operation request is a file reading request, and the method comprises the steps of calling a file opening function, inquiring a metadata base to determine whether a first physical path corresponding to a first logic path exists, calling the file reading function in response to receiving the file reading request if the first physical path exists, acquiring file data corresponding to the first physical path through the file reading function, and sending the file data to a user program through a virtual file system. According to the embodiment of the application, the method comprises the steps of responding to a received file screening request, inquiring a metadata base according to preset screening conditions, obtaining a plurality of pieces of matching metadata, wherein the plurality of pieces of matching metadata have different levels of physical paths, modifying a logic father path of the plurality of pieces of matching metadata into the same logic path, extracting file names of the plurality of pieces of modified matching metadata, and sending the file names to a user program through a virtual file system to form a screening catalog list. According to an embodiment of the application, the method comprises creating at least one new piece of metadata in a metadata base in response to receiving an alias creation instruction for a first logical path, the first logical path corresponding to initial metadata, the physical path of the initial metadata being a first physical path, wherein the physical path of the at least one new piece of metadata is the first physical path, the logica