Search

CN-115481096-B - Metadata processing method, device, equipment and medium

CN115481096BCN 115481096 BCN115481096 BCN 115481096BCN-115481096-B

Abstract

The application discloses a metadata processing method, a device, equipment and a medium, which are applied to a distributed file system, wherein the method comprises the steps of receiving a file operation request; and processing the metadata according to the type of the metadata operation and the file type, wherein the metadata is stored based on a key value pair, the key comprises a unique number of a parent file and a file name of a sub file, and the value is the file information of the sub file. The method of the application can improve the efficiency of metadata processing and accurately recover the data under abnormal conditions.

Inventors

  • FANG RAN
  • WU GAN

Assignees

  • 深圳云创天地信息技术有限公司

Dates

Publication Date
20260508
Application Date
20220805

Claims (12)

  1. 1. A metadata processing method applied to a distributed file system, the method comprising: Receiving a file operation request; Responding to the file operation request, and analyzing metadata operation; Processing metadata according to the type of the metadata operation and the file type, wherein the metadata is stored based on a key value pair, the key comprises a unique number of a parent file and a file name of a child file, and the value is file information of the child file; And writing the file information into a metadata cache module after the file operation, updating the file length in the metadata cache module for the file which is being read and written, and periodically refreshing the data in the metadata cache module into a metadata service module after the file length is increased to a preset value.
  2. 2. The method of claim 1, wherein when the subfile is a folder, Forming a second key according to the unique number of the subfile; and forming a second value according to the unique number of the parent file and the file name of the child file.
  3. 3. The method for metadata processing according to claim 2, wherein when the subfile is a hard link file, And forming a third key according to the unique number of the sub-file, and forming a third value according to the file information of the hard link file.
  4. 4. The metadata processing method according to claim 1, wherein the value includes soft link path information when the subfile is a soft link file.
  5. 5. The method according to claim 1, wherein when the type of the metadata operation is a renamed file, the method comprises changing the record of the sub-file into a hard link form, inserting the renamed sub-file record, and deleting the sub-file.
  6. 6. The method according to claim 2, wherein when the type of the metadata operation is creation of a folder, the method includes forming the key after forming the value.
  7. 7. A metadata processing method according to claim 3, wherein when the type of metadata operation is creation of a hard link file, the method comprises modifying hard link records in a source file and a destination file first, modifying kv records of the source file again, and inserting kv records of the destination file last.
  8. 8. A method of metadata processing according to claim 1, comprising writing data into a metadata cache and updating the data in the metadata cache to a kv record according to the length of the written file.
  9. 9. A metadata management apparatus for a distributed file system, the apparatus comprising: the file service module is used for receiving a file operation request from the client and analyzing metadata operation; the metadata service module is used for processing metadata according to the type of metadata operation and the file type, wherein the metadata is stored based on key value pairs, the key comprises a unique number of a parent file and a file name of a child file, and the value is file information of the child file; and writing the file information into a metadata cache module after the file operation, updating the file length in the metadata cache module for the file which is being read and written, and periodically refreshing the data in the metadata cache module into a metadata service module after the file length is increased to a preset value.
  10. 10. The device according to claim 9, wherein the file service module further comprises a metadata cache module for writing data into a metadata cache and updating the data in the metadata cache to kv record according to the length of the written file.
  11. 11. An electronic device comprising a memory storing computer executable instructions and a processor, which when executed by the processor, cause the device to implement the method of any one of claims 1 to 8.
  12. 12. A computer readable medium, characterized in that the medium stores one or more programs executable by one or more processors to implement the method of any of claims 1 to 8.

Description

Metadata processing method, device, equipment and medium Technical Field The present invention relates to the field of computer technologies, and in particular, to a metadata processing method, apparatus, device, and medium. Background In a common distributed file system architecture, file data is stored separately from metadata. In the application of the distributed file system, a user can frequently perform operations such as reading and writing a large number of files, inquiring catalogues and the like, and a large number of metadata access and record modification can be generated. In the related art, in order to repeatedly operate on the same file, the file is directly positioned by using a relatively fixed index to improve efficiency of searching and modifying file information. However, the operation of these methods is cumbersome and the data may not be easily restored in abnormal situations. Disclosure of Invention The metadata processing method, the device, the equipment and the medium provided by the application can improve the metadata processing efficiency and accurately recover the data under the abnormal condition. The first aspect of the application discloses a metadata processing method which is applied to a distributed file system and comprises the steps of receiving a file operation request, responding to the file operation request, analyzing metadata operation, and processing metadata according to the type of the metadata operation and the file type, wherein the metadata is stored based on a key value pair, the key comprises a unique number of a parent file and a file name of a child file, and the value is file information of the child file. In a possible implementation manner of the first aspect, when the sub file is a folder, a second key is formed according to the unique number of the sub file, and a second value is formed according to the unique number of the parent file and the file name of the sub file. In a possible implementation of the first aspect, when the subfile is a hard link file, the value includes a hard link flag, and a third key is formed according to a unique number of the subfile, and a third value is formed according to file information of the hard link file. In a possible implementation of the first aspect, when the subfile is a soft link file, the value includes soft link path information. In a possible implementation manner of the first aspect, when the type of the metadata operation is a renamed file, the method includes changing a record of the sub-file into a hard link form, inserting the renamed sub-file record, and deleting the sub-file. In a possible implementation of the first aspect, when the type of the metadata operation is creation of a folder, the method includes forming the key after forming the value. In a possible implementation of the first aspect, when the type of the metadata operation is to create a hard link file, the method includes modifying hard link records in a source file and a destination file, modifying a kv record of the source file, and inserting a kv record of the destination file. In a possible implementation of the first aspect, the method includes writing data into a metadata cache, and updating the data in the metadata cache to a kv record according to a length of a write file. The application discloses a metadata management device of a distributed file system, which comprises a file service module and a metadata service module, wherein the file service module is used for receiving a file operation request from a client and analyzing metadata operation, the metadata service module is used for processing metadata according to the type of the metadata operation and the file type, the metadata is stored based on a key value pair, the key comprises a unique number of a parent file and a file name of a sub-file, and the value is file information of the sub-file. In a possible implementation manner of the second aspect, the file service module further includes a metadata cache module, configured to write data into a metadata cache, and update data in the metadata cache to a kv record according to a length of a written file. A third aspect of the application discloses an electronic device comprising a memory storing computer executable instructions and a processor, which when executed by the processor, causes the device to carry out the method of the embodiment of the first aspect described above. A fourth aspect of the application discloses a computer readable storage medium storing one or more computer programs for execution by one or more processors such that the processors perform the method of the embodiments of the first aspect described above. According to the metadata processing method, device, equipment and medium, after receiving the metadata operation request, metadata is operated according to the type of operation and the type of the file to be processed, wherein the metadata is stored in a kv form, file information of the su