CN-122027720-A - Remote procedure call data transmission method and device

CN122027720ACN 122027720 ACN122027720 ACN 122027720ACN-122027720-A

Abstract

The embodiment of the application discloses a data transmission method and device for remote procedure call. The method comprises the steps of optimizing a memory layout of data to be sent to be aligned with a target network transmission format, generating serialized data based on the aligned memory layout, wherein the serialized data carry information indicating a data structure of the serialized data, and transmitting the serialized data to a receiving end by utilizing a kernel bypass network transmission technology so that the receiving end can analyze the serialized data based on the information indicating the data structure of the serialized data, wherein the kernel bypass network transmission technology comprises a remote direct memory access technology or a user mode transmission control protocol. The application can realize zero-copy serialization and high-efficiency data transmission, thereby improving communication performance.

Inventors

KONG YU
WANG JING
LIANG XIAO
XIE MINHUI

Assignees

北京达佳互联信息技术有限公司
中国人民大学

Dates

Publication Date: 20260512
Application Date: 20260212

Claims (15)

1. The data transmission method for remote procedure call is characterized by being applied to a transmitting end, wherein a communication connection is established between the transmitting end and a receiving end, and the method comprises the following steps: optimizing the memory layout of the data to be transmitted to be aligned with the transmission format of the target network; Generating serialized data based on the aligned memory layout, wherein the serialized data carries information indicating a data structure of the serialized data; And transmitting the serialized data to a receiving end by using a kernel bypass type network transmission technology so that the receiving end can analyze the serialized data based on the information indicating the data structure of the serialized data, wherein the kernel bypass type network transmission technology comprises a remote direct memory access technology or a user mode transmission control protocol.
2. The method of claim 1, wherein optimizing the memory layout of the data to be transmitted to be aligned with the target network transport format comprises: determining a Schema definition corresponding to a target network transmission format, wherein the Schema definition at least comprises sequence and type information of fields; And based on the Schema definition, at least one of field rearrangement and hole elimination is performed on the memory layout of the data to be sent, so that the optimized memory layout is aligned with the target network transmission format.
3. The method of claim 1, wherein generating the serialized data based on the aligned memory layout comprises: Generating an address area and a data head based on the aligned memory layout, wherein the address area comprises offset information of each field corresponding to the aligned memory layout, and the data head comprises information indicating a data structure of the serialized data, and the data structure of the serialized data is the aligned memory layout; and obtaining the serialized data based on the data head, the address area and the value area, wherein the value area corresponds to the data after the memory layout optimization.
4. The method of claim 1, further comprising at least one of: when the sending end and the receiving end establish the communication connection, negotiating to take a remote direct memory access technology or a user mode transmission control protocol as the kernel bypass network transmission technology based on network environment and/or application characteristics; in response to detecting a change in the network environment, renegotiating to use a remote direct memory access technique or a user mode transmission control protocol as the kernel bypass network transmission technique based on the changed network environment.
5. The method of claim 4, wherein negotiating a remote direct memory access technology or a user mode transmission control protocol as the kernel bypass network transmission technology comprises: Negotiating to take the remote direct memory access technology as the kernel bypass network transmission technology if the network environment meets the network environment requirement of the remote direct memory access technology, otherwise negotiating to take the user mode transmission control protocol as the kernel bypass network transmission technology, or Negotiating to use a remote direct memory access technology as the kernel bypass network transmission technology if the network delay requirement of the application feature is higher than a preset delay requirement, otherwise negotiating to use a user mode transmission control protocol as the kernel bypass network transmission technology, or If the network environment meets the network environment requirement of the remote direct memory access technology and the network delay requirement of the application characteristic is higher than the preset delay requirement, negotiating to take the remote direct memory access technology as the kernel bypass network transmission technology, otherwise negotiating to take the user mode transmission control protocol as the kernel bypass network transmission technology.
6. The method of claim 1, further comprising, prior to optimizing the memory layout of the data to be transmitted to align with the target network transport format: the transmitting end negotiates with the receiving end about the adopted target network transmission format, wherein the target network transmission format comprises a RAW format or FlatKV format.
7. The method according to any one of claims 1 to 6, wherein the transmitting end is a server end and the receiving end is a client end, the method further comprising: When the sending end and the receiving end establish the communication connection, a reverse command channel is established; and sending a control command to the client by using the reverse command channel, wherein the control command comprises a service switching command for indicating to switch to a new server side.
8. A data transmission method for remote procedure call, which is applied to a receiving end, wherein a communication connection is established between the receiving end and a transmitting end, the method comprising: receiving the serialized data transmitted by the transmitting end by using a kernel bypass network transmission technology, wherein the memory layout of the serialized data is aligned with a target network transmission format and is generated based on the aligned memory layout; analyzing the serialized data based on the information, carried by the serialized data, indicating the data structure of the serialized data, wherein the kernel bypass network transmission technology comprises a remote direct memory access technology or a user mode transmission control protocol.
9. The method of claim 8, wherein the information indicating the data structure of the serialized data includes Schema version information, wherein parsing the serialized data based on the information indicating the data structure of the serialized data carried by the serialized data includes: Acquiring the Schema version information from the data header of the serialized data; Searching a corresponding Schema definition based on the Schema version information, wherein the Schema definition at least comprises sequence and type information of fields; Based on the sequence of the fields, respectively acquiring offset information of each field in a value area of the serialized data from an address area of the serialized data; Based on the offset information and the type information of each field, reading field values from the value areas according to the sequence of the fields and writing the field values into a memory.
10. The method according to claim 8 or 9, wherein the transmitting end is a server end and the receiving end is a client end, the method further comprising: When the receiving end and the transmitting end establish the communication connection, a reverse command channel is established; receiving a control command sent by the server by utilizing the reverse command channel, wherein the control command comprises a service switching command for indicating to switch to a new server; And switching the communication connection established with the server to the communication connection established with the new server.
11. A data transmission device for remote procedure call, applied to a transmitting end, wherein a communication connection is established between the transmitting end and a receiving end, the device comprising: The memory optimizing unit is configured to optimize the memory layout of the data to be transmitted to be aligned with the transmission format of the target network; a serialization unit configured to generate serialization data based on the aligned memory layout, the serialization data carrying information indicating a data structure of the serialization data; And the network transmission unit is configured to transmit the serialized data to a receiving end by using a kernel bypass type network transmission technology so that the receiving end can analyze the serialized data based on the information indicating the data structure of the serialized data, wherein the kernel bypass type network transmission technology comprises a remote direct memory access technology or a user mode transmission control protocol.
12. A data transmission device for remote procedure call, applied to a receiving end, wherein a communication connection is established between the receiving end and a transmitting end, the method comprising: the network transmission unit is configured to receive the serialized data transmitted by the transmitting end by utilizing a kernel bypass network transmission technology, the memory layout of the serialized data is aligned with a target network transmission format, and the serialized data is generated based on the aligned memory layout; The data analysis unit is configured to analyze the serialized data based on the information, carried by the serialized data, indicating the data structure of the serialized data, wherein the kernel bypass network transmission technology comprises a remote direct memory access technology or a user mode transmission control protocol.
13. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the method of any of claims 1 to 10.
14. An electronic device, comprising: One or more processors, and A memory associated with the one or more processors for storing program instructions that, when read for execution by the one or more processors, perform the steps of the method of any of claims 1 to 10.
15. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 10.

Description

Remote procedure call data transmission method and device Technical Field The present application relates to the field of remote procedure call, and in particular, to a method and apparatus for data transmission of remote procedure call. Background Remote procedure calls (Remote Procedure Call, RPC) are the primary way distributed system communications allows a sender (e.g., server side) to call a receiver (e.g., client side) procedure like a local procedure. However, there is a significant bottleneck in the RPC framework in the high performance scenario at present, and resource utilization and response delay become problems to be solved urgently. Disclosure of Invention In view of the above, the present application provides a method and apparatus for transmitting data for remote procedure call, so as to improve the resource utilization and reduce the response delay. The application provides the following scheme: In a first aspect, a data transmission method for remote procedure call is provided, and the method is applied to a transmitting end, and a communication connection is established between the transmitting end and a receiving end, and includes: optimizing the memory layout of the data to be transmitted to be aligned with the transmission format of the target network; Generating serialized data based on the aligned memory layout, wherein the serialized data carries information indicating a data structure of the serialized data; And transmitting the serialized data to a receiving end by using a kernel bypass type network transmission technology so that the receiving end can analyze the serialized data based on the information indicating the data structure of the serialized data, wherein the kernel bypass type network transmission technology comprises a remote direct memory access technology or a user mode transmission control protocol. Optionally, the optimizing the memory layout of the data to be sent to be aligned with the target network transmission format includes: determining a Schema definition corresponding to a target network transmission format, wherein the Schema definition at least comprises sequence and type information of fields; And based on the Schema definition, at least one of field rearrangement and hole elimination is performed on the memory layout of the data to be sent, so that the optimized memory layout is aligned with the target network transmission format. Optionally, generating the serialized data based on the aligned memory layout includes: Generating an address area and a data head based on the aligned memory layout, wherein the address area comprises offset information of each field corresponding to the aligned memory layout, and the data head comprises information indicating a data structure of the serialized data, and the data structure of the serialized data is the aligned memory layout; and obtaining the serialized data based on the data head, the address area and the value area, wherein the value area corresponds to the data after the memory layout optimization. Optionally, the method further comprises at least one of: when the sending end and the receiving end establish the communication connection, negotiating to take a remote direct memory access technology or a user mode transmission control protocol as the kernel bypass network transmission technology based on network environment and/or application characteristics; in response to detecting a change in the network environment, renegotiating to use a remote direct memory access technique or a user mode transmission control protocol as the kernel bypass network transmission technique based on the changed network environment. Optionally, the negotiating uses a remote direct memory access technology or a user mode transmission control protocol as the kernel bypass network transmission technology, including: Negotiating to take the remote direct memory access technology as the kernel bypass network transmission technology if the network environment meets the network environment requirement of the remote direct memory access technology, otherwise negotiating to take the user mode transmission control protocol as the kernel bypass network transmission technology, or Negotiating to use a remote direct memory access technology as the kernel bypass network transmission technology if the network delay requirement of the application feature is higher than a preset delay requirement, otherwise negotiating to use a user mode transmission control protocol as the kernel bypass network transmission technology, or If the network environment meets the network environment requirement of the remote direct memory access technology and the network delay requirement of the application characteristic is higher than the preset delay requirement, negotiating to take the remote direct memory access technology as the kernel bypass network transmission technology, otherwise negotiating to take the user mode transmission control protocol as the