Search

CN-122019607-A - Method and system for storing SPDK metadata

CN122019607ACN 122019607 ACN122019607 ACN 122019607ACN-122019607-A

Abstract

The embodiment of the invention provides a method and a storage system for storing SPDK metadata, wherein the method comprises the steps of storing metadata in a built-in metadata storage mode in a compression disc where a compression function of a storage performance development kit SPDK is located, determining a storage position of corresponding metadata in a metadata storage space according to carried first parameter information under the condition that any reading request of a user is received, searching the storage position of the compressed data in the corresponding metadata according to the storage position of the metadata, reading the corresponding metadata according to second parameter information carried by a writing request under the condition that any writing request of the user is received, determining the storage position of the compressed data according to the metadata, storing the metadata, and writing the corresponding metadata in the metadata storage space according to the storage position of the compressed data. The metadata is directly written into the memory without real-time solidification, so that the CPU occupation is reduced.

Inventors

  • Wang yalong
  • WANG WEI

Assignees

  • 新浪技术（中国）有限公司

Dates

Publication Date
20260512
Application Date
20251231

Claims (11)

  1. 1. A method of storing SPDK metadata, comprising: storing metadata in a compression disk where a compression function of the storage performance development kit SPDK is located by adopting a built-in metadata storage mode, wherein the built-in metadata storage mode comprises the steps of applying a plurality of memory pages with the same size as a storage block space in the compression disk as metadata storage spaces, and determining the size of the storage block space according to the metadata type to be stored in SPDK; Under the condition that any reading request of a user is received, determining a storage position of corresponding metadata in a metadata storage space according to first parameter information carried by the reading request, and searching a storage position of compressed data in the corresponding metadata according to the storage position of the metadata, wherein the storage position of the compressed data is used for acquiring the compressed data for the user and decompressing the compressed data to be returned to the user; And under the condition that any write request of a user is received, corresponding metadata is read according to second parameter information carried by the write request, the storage position of compressed data to be written in by the write request is determined and stored according to the read metadata, and corresponding metadata is written in the metadata storage space according to the storage position of the compressed data.
  2. 2. The method of storing SPDK compressed disc metadata as in claim 1 wherein the storage block space size is determined according to the type of metadata to be stored at SPDK, comprising: setting a corresponding metadata block, namely a Superblock block, a Logicalmap block or a Chunkmap block, for each storage block according to a metadata type to be stored SPDK, wherein the metadata type at least comprises volume attribute information, address mapping information and a data storage position, and the metadata type comprises the following components: the Superblock block is used for storing volume attribute information, wherein the volume attribute information at least comprises a unique identifier of a compression volume in a compression disk and a storage unit size used for storing compression data; logicalmap blocks are used to store address mapping information, including mapping information of user addresses to Chunkmap within Chunkmap blocks; Chunkmap within Chunkmap blocks is used to record data storage locations that are storage locations for compressed data after the original data is compressed.
  3. 3. The method for storing SPDK compressed disc metadata according to claim 2, wherein in the case of receiving any one of the read requests of the user, determining the storage location of the corresponding metadata according to the first parameter information carried by the read request, searching the storage location of the compressed data in the corresponding metadata according to the storage location of the metadata, and reading the data for the user according to the storage location of the compressed data includes: under the condition that any reading request of a user is received, calculating according to first parameter information carried by the reading request to obtain a storage block sequence number of metadata in a compression disc, wherein the first parameter information comprises offset and length; Reading the block offset position of the Logicalmap block corresponding to the Logicalmap block sequence number according to the Logicalmap block sequence number and the block offset in the storage block corresponding to the storage block sequence number to obtain a Chunkmap sequence number; Calculating Chunkmap blocks and intra-block offset according to the Chunkmap serial numbers, and reading the intra-block offset position of Chunkmap blocks to obtain Chunkmap; And acquiring a storage position of compressed data after the original data is compressed in the Chunkmap, acquiring the compressed data from the storage position of the compressed data, and decompressing the compressed data to return to a user.
  4. 4. The method for storing SPDK compressed disc metadata according to claim 2, wherein in the case of receiving any write request from a user, reading corresponding metadata according to second parameter information carried by the write request, determining a storage location of compressed data to be written by the write request according to the read metadata, and storing the compressed data, and writing corresponding metadata in the metadata storage space according to the storage location of the compressed data, including: Under the condition that any write request of a user is received, calculating according to second parameter information carried by the write request to obtain a storage block sequence number of metadata in a compression disc, wherein the second parameter information comprises offset and length; According to Logicalmap block serial numbers and intra-block offsets in the storage blocks corresponding to the storage block serial numbers, if the value of Logicalmap at the intra-block offset position in the Logicalmap block corresponding to the Logicalmap block serial numbers is null, confirming that the data corresponding to the write request is written for the first time, and if the value of Logicalmap at the intra-block offset position in the Logicalmap block corresponding to the Logicalmap block serial numbers is not null, confirming that the data corresponding to the write request is written for updating; For the case that the data corresponding to the writing request is written for the first time, compressing the data to be written by the writing request to form compressed data, taking an idle position in a storage unit as a storage position of the compressed data, recording the storage position on the idle Chunkmap, and storing the corresponding Chunkmap serial number to Logicalmap with a value of idle in Logicalmap blocks calculated by a user request; For the case that the data corresponding to the writing request is written into as updated writing, reading the corresponding Chunkmap according to the Chunkmap serial number recorded on the Logicalmap block, analyzing Chunkmap, reading the corresponding compressed data according to the storage position of the compressed data recorded in the Chunkmap, decompressing the read compressed data to restore original data, adopting the new data corresponding to the writing request to cover the original data, determining the free Chunkmap block and the storage position of the compressed data obtained by compressing the new data through the resource bitmap, firstly storing the compressed data obtained by compressing the new data on the new storage position in the storage unit, recording the new storage position of the compressed data obtained by compressing the new data on the Chunkmap where the Chunkmap is written into the disk, and finally writing the Chunkmap serial number into Logicalmap where the value in the Logicalmap block in the storage block where the user request is located is empty.
  5. 5. The method of storing SPDK compressed disc metadata as in claim 2, further comprising: setting a through buffer device for the metadata storage space correspondingly, wherein in the through buffer device, a buffer unit is correspondingly set for each metadata block, and a read waiting queue and a write waiting queue are arranged on each buffer unit and used for respectively mounting a read request and a write request which need to access the metadata block at the same time; before any metadata block is read, checking whether a corresponding cache unit exists in the direct-connection cache device, and if the corresponding cache unit exists in the direct-connection cache device, directly accessing the corresponding cache unit; If no corresponding cache unit exists in the direct cache device, reading the metadata storage space in the compression disk; If a read request for accessing the metadata block is written into a read waiting queue or the write request is added into the write waiting queue in the process of reading the metadata storage space in the compression disk, after the reading is finished, informing the read result to the same request in the read waiting queue or the write waiting queue one by one, and simultaneously storing the read metadata block into a cache unit in the direct cache device.
  6. 6. The method for storing SPDK compressed disc metadata as in claim 2 further comprising the step of reloading the metadata storage space after SPDK process restart or a local machine restart, specifically comprising: after SPDK processes are restarted or the machine where the processes are located is restarted, superblock blocks in the compressed disk are read, volume attribute information is checked, whether a built-in metadata storage mode is adopted in SPDK is determined, if the built-in metadata storage mode is determined to be adopted, a plurality of memory pages are required to be applied from a memory to serve as metadata caches and storage blocks for data reading and writing with the disk, and a control structure for managing each memory page is applied; Logicalmap blocks and Chunkmap blocks located after the Superblock block are read, and the Chunkmap bitmap and the resource bitmap are restored in sequence.
  7. 7. A storage system for SPDK metadata, comprising: the metadata storage space is arranged in a compression disc where the compression function of the storage performance development kit SPDK is located, and is obtained from memory pages with the same size applied in the compression disc, and the size of the storage block space is determined according to the metadata type to be stored in SPDK and is used for storing the metadata in the compression disc by adopting a built-in metadata storage mode; The metadata reading unit is used for determining the storage position of corresponding metadata in a metadata storage space according to first parameter information carried by a read request under the condition that any read request of a user is received, searching the storage position of compressed data in the corresponding metadata according to the storage position of the metadata, wherein the storage position of the compressed data is used for acquiring the compressed data for the user and decompressing the compressed data and returning the compressed data to the user; And the metadata writing unit is used for reading corresponding metadata according to the second parameter information carried by the writing request under the condition that any writing request of a user is received, determining the storage position of the compressed data to be written in by the writing request according to the read metadata, storing the storage position, and writing corresponding metadata in the metadata storage space according to the storage position of the compressed data.
  8. 8. The storage system of SPDK metadata as set forth in claim 7 wherein said storage blocks comprise metadata blocks Superblock, logicalmap, or Chunkmap blocks, for determining a metadata type to be stored according to SPDK, said metadata type comprising at least volume attribute information, address mapping information, and data storage locations, wherein: The Superblock block is used for storing volume attribute information, and the volume attribute information at least comprises a unique identifier of a compression volume in a compression disk and a storage unit size used for storing compression data; Logicalmap blocks for storing address mapping information including mapping information of user addresses to Chunkmap within Chunkmap blocks; Chunkmap block Chunkmap is used to record data storage locations that are storage locations for compressed data after the original data has been compressed.
  9. 9. The SPDK metadata storage system as in claim 8 further comprising a pass-through caching device and a metadata synchronization unit, wherein: the direct-connection caching device is correspondingly arranged for the metadata storage space, a caching unit is correspondingly arranged for each metadata block, and a read waiting queue and a write waiting queue are arranged on the caching unit and used for respectively mounting read requests and write requests which need to access the metadata block at the same time; The metadata reading unit is further configured to, before any metadata block is read, check whether a corresponding cache unit exists in the through cache device, and if the corresponding cache unit exists in the through cache device, directly access the corresponding cache unit; And the metadata synchronization unit is used for notifying the read result to the same request in the read waiting queue or the write waiting queue one by one after the reading is finished if the read request for accessing the metadata block is written into the read waiting queue or the write request is added into the write waiting queue in the process of reading the metadata storage space in the compression disk, and storing the read metadata block into the cache unit in the direct cache device.
  10. 10. A computer readable storage medium storing one or more programs, which when executed by a computer device, cause the computer device to perform the method of storing SPDK compact disc metadata of any one of claims 1-6.
  11. 11. A computer device, comprising: And a memory arranged to store computer executable instructions that when executed cause the processor to perform the method of storing SPDK compressed disc metadata of any one of claims 1 to 6.

Description

Method and system for storing SPDK metadata Technical Field The invention relates to the field of metadata storage, in particular to a method and a storage system for storing SPDK metadata. Background SPDK (Storage Performance Development Kit) is an open source software library designed to provide a high performance block storage solution that allows developers to bypass the traditional operating system software stack, interact directly with the hardware, and thus achieve higher I/O throughput and lower latency. The REDUCE and VBDEV _ COMPRESS modules in SPDK provide compression functions, which can create a compressed disk on a storage device with reduced functions, thus realizing intensive use of storage resources. In carrying out the present invention, the applicant has found that at least the following problems exist in the prior art: SPDK require an additional metadata storage medium, where persistent memory is preferably used to achieve optimal metadata performance. A file system can also be established on a common disk to store metadata, and files can be created and mapped to a memory for use. Although the optimal metadata storage medium for SPDK compression functions is persistent memory, the use of this hardware is limited. So, at present, a file system can only be established based on a common magnetic disk, and a file is used for storing compressed disc metadata, but the metadata storage mode has the following technical problems: Mapping files in the file system into SPDK processes requires occupying the same amount of physical memory as the size of metadata space, and the memory consumption is large. Disclosure of Invention The embodiment of the invention provides a method and a system for storing SPDK metadata, which can solve the technical problem that the memory occupied by the metadata stored in a file system established by adopting a common disk in the prior art is more. To achieve the above object, in a first aspect, an embodiment of the present invention provides a method for storing SPDK metadata, including: storing metadata in a compression disk where a compression function of the storage performance development kit SPDK is located by adopting a built-in metadata storage mode, wherein the built-in metadata storage mode comprises the steps of applying a plurality of memory pages with the same size as a storage block space in the compression disk as metadata storage spaces, and determining the size of the storage block space according to the metadata type to be stored in SPDK; Under the condition that any reading request of a user is received, determining a storage position of corresponding metadata in a metadata storage space according to first parameter information carried by the reading request, and searching a storage position of compressed data in the corresponding metadata according to the storage position of the metadata, wherein the storage position of the compressed data is used for acquiring the compressed data for the user and decompressing the compressed data to be returned to the user; And under the condition that any write request of a user is received, corresponding metadata is read according to second parameter information carried by the write request, the storage position of compressed data to be written in by the write request is determined and stored according to the read metadata, and corresponding metadata is written in the metadata storage space according to the storage position of the compressed data. In a second aspect, an embodiment of the present invention provides a storage system for SPDK metadata, including: the metadata storage space is arranged in a compression disc where the compression function of the storage performance development kit SPDK is located, and is obtained from memory pages with the same size applied in the compression disc, and the size of the storage block space is determined according to the metadata type to be stored in SPDK and is used for storing the metadata in the compression disc by adopting a built-in metadata storage mode; The metadata reading unit is used for determining the storage position of corresponding metadata in a metadata storage space according to first parameter information carried by a read request under the condition that any read request of a user is received, searching the storage position of compressed data in the corresponding metadata according to the storage position of the metadata, wherein the storage position of the compressed data is used for acquiring the compressed data for the user and decompressing the compressed data and returning the compressed data to the user; And the metadata writing unit is used for reading corresponding metadata according to the second parameter information carried by the writing request under the condition that any writing request of a user is received, determining the storage position of the compressed data to be written in by the writing request according to the read metadata, s