CN-122027600-A - Data access method, device, system, electronic equipment and storage medium

CN122027600ACN 122027600 ACN122027600 ACN 122027600ACN-122027600-A

Abstract

The disclosure provides a data access method, a device, a system, electronic equipment and a storage medium, and relates to the technical field of high-performance computing and cloud storage acceleration. The method comprises the steps of intercepting a simple storage service S3 object access request, registering a target memory address corresponding to the S3 object access request in a storage space, establishing Remote Direct Memory Access (RDMA) connection with a storage server, and transmitting the target object of the S3 object access request in the storage server to the target memory address through the RDMA connection. The scheme can reduce the data transmission delay and improve the data transmission efficiency.

Inventors

Request for anonymity
Request for anonymity
Request for anonymity
Request for anonymity
Request for anonymity

Assignees

摩尔线程智能科技(北京)股份有限公司

Dates

Publication Date: 20260512
Application Date: 20260130

Claims (10)

1. A method of data access, the method comprising: Intercepting a simple storage service S3 object access request; Registering a target memory address corresponding to the S3 object access request in a storage space; And transmitting the target object of the S3 object access request in the storage server to the target memory address through a Remote Direct Memory Access (RDMA) connection with the storage server.
2. The data access method according to claim 1, wherein after intercepting the simple storage service S3 object access request, the method further comprises: Acquiring metadata of a target object to be accessed by the S3 object access request; and if the metadata indicates that the target object supports RDMA transmission, executing the step of registering the target memory address corresponding to the S3 object access request in the storage space.
3. The method according to claim 2, wherein the obtaining metadata of the target object to be accessed by the S3 object access request includes: analyzing the S3 object access request, and determining a target object to be accessed by the S3 object access request; Inquiring whether metadata of the target object exists in a cache; And if the metadata of the target object does not exist in the cache, acquiring the metadata of the target object from a storage server.
4. The method according to claim 2, wherein the step of registering the target memory address corresponding to the S3 object access request in the storage space is performed if the metadata indicates that the target object supports RDMA transfer, comprising: And if the metadata indicates that the target object supports RDMA transmission and the size of the target object is larger than a specified size threshold, executing the step of registering the target memory address corresponding to the S3 object access request in the storage space.
5. The data access method according to any one of claims 1 to 4, wherein intercepting the simple storage service S3 object access request comprises: intercepting a simple storage service S3 object access request of an application program; the method further comprises the steps of: and under the condition that the transmission of the target object is completed, generating an S3 object access completion notification, and calling back the S3 object access completion notification to the application program.
6. The method according to claim 5, wherein callback the S3 object access completion notification to the application program comprises: storing the S3 object access completion notification into a completion queue; If the polling mode is adopted, carrying out polling detection on the completion queue, and calling back the S3 object access completion notification to the application program after the polling detection is completed; if the interrupt mode is adopted, when the S3 object access completion notification stored in the completion queue is detected, interrupt detection is carried out, and the S3 object access completion notification is called back to the application program.
7. A data access device, comprising: The request interception module is used for intercepting the simple storage service S3 object access request; the memory address registration module is used for registering a target memory address corresponding to the S3 object access request in a storage space; And the data transmission module is used for transmitting the target object of the S3 object access request in the storage server to the target memory address through Remote Direct Memory Access (RDMA) connection with the storage server.
8. A data access system, comprising: The application program is run on the upper layer of the computing node and is used for initiating a simple storage service S3 object access request; means disposed at a layer of said computing node, said means comprising the data access means of claim 7; The communication interface is arranged at the bottom layer of the computing node and is used for providing a data path of the RDMA connection and supporting control plane and data plane interaction between the data access device and the storage server; The storage server is configured to store the target object and support transfer of the target object over the RDMA connection.
9. An electronic device, comprising: Processor, and A memory having stored thereon computer readable instructions which when executed by the processor implement the data access method of any of claims 1 to 6.
10. A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the data access method according to any one of claims 1 to 6.

Description

Data access method, device, system, electronic equipment and storage medium Technical Field The disclosure relates to the technical field of high-performance computing and cloud storage acceleration, in particular to a data access method, a device, a system, electronic equipment and a storage medium. Background With the deep fusion of cloud computing and artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) technology, high-performance computing based on GPU (Graphics Processing Unit, graphics processor) has been widely applied to multiple scenarios such as machine learning reasoning, scientific computing, video processing, large model service, etc. In these scenarios, input data (e.g., images, speech, model parameters, or profiles) is typically stored in the form of objects in a cloud-native object storage system. The conventional method generally uses an S3 (Simple Storage Service ) interface to develop online applications, and the storage of the S3 interface has great advantages in terms of usability, serviceability, reliability and the like. However, at present, all S3 storage can only be transmitted based on a TCP (Transmission Control Protocol ) network, which has the problem of higher data transmission delay. It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art. Disclosure of Invention An object of an embodiment of the present disclosure is to provide a data access method, apparatus, system, electronic device, and computer readable storage medium, which can reduce data transmission delay and improve data transmission efficiency. Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure. According to a first aspect of embodiments of the present disclosure, there is provided a data access method, the method comprising: Intercepting a simple storage service S3 object access request; Registering a target memory address corresponding to the S3 object access request in a storage space; And transmitting the target object of the S3 object access request in the storage server to the target memory address through a Remote Direct Memory Access (RDMA) connection with the storage server. In some example embodiments of the present disclosure, based on the foregoing solution, after intercepting the simple storage service S3 object access request, the method further includes: Acquiring metadata of a target object to be accessed by the S3 object access request; and if the metadata indicates that the target object supports RDMA transmission, executing the step of registering the target memory address corresponding to the S3 object access request in the storage space. In some example embodiments of the present disclosure, based on the foregoing solution, the obtaining metadata of the target object to be accessed by the S3 object access request includes: analyzing the S3 object access request, and determining a target object to be accessed by the S3 object access request; Inquiring whether metadata of the target object exists in a cache; And if the metadata of the target object does not exist in the cache, acquiring the metadata of the target object from a storage server. In some example embodiments of the present disclosure, based on the foregoing solution, the step of performing the registering, in the storage space, the target memory address corresponding to the S3 object access request if the metadata indicates that the target object supports RDMA transfer includes: And if the metadata indicates that the target object supports RDMA transmission and the size of the target object is larger than a specified size threshold, executing the step of registering the target memory address corresponding to the S3 object access request in the storage space. In some example embodiments of the present disclosure, based on the foregoing solution, the intercepting the simple storage service S3 object access request includes: intercepting a simple storage service S3 object access request of an application program; the method further comprises the steps of: and under the condition that the transmission of the target object is completed, generating an S3 object access completion notification, and calling back the S3 object access completion notification to the application program. In some example embodiments of the present disclosure, based on the foregoing solution, the callback of the S3 object access completion notification to the application includes: storing the S3 object access completion notification into a completion queue; If the polling mode is adopted, carrying out polling detection on the completion queue, and calling back the S3 object access completion notification to the application progra