Search

CN-115454677-B - Data transmission method and system

CN115454677BCN 115454677 BCN115454677 BCN 115454677BCN-115454677-B

Abstract

The application discloses a data transmission method and system, wherein the method comprises the steps that a publisher registers a topic by creating a shared memory and initializing metadata, the publisher stores data to be transmitted into the shared memory and the metadata, a subscriber subscribes the topic and maps the metadata under the topic to a memory space of the subscriber, and the subscriber takes out the data to be transmitted in the memory space through deserialization according to the metadata. The application realizes high-performance data transmission among modules in the same computing unit based on the technologies of shared memory, non-lock data structure, zero copy and the like, and has the advantages of high throughput rate, low delay, no blocking, zero copy, no single point fault and the like.

Inventors

  • HE HUAHAI

Assignees

  • 广州文远知行科技有限公司

Dates

Publication Date
20260508
Application Date
20220926

Claims (7)

  1. 1. A data transmission method, comprising: the publisher registers topic by creating a shared memory and initializing metadata, including: the publisher creates a shared memory and takes the shared memory as a shared memory ring buffer area under a topic, wherein the shared memory ring buffer area is provided with a head pointer and a tail pointer; initializing metadata by the publisher, wherein the metadata comprises a message metadata array with head and tail pointers, a pre-stored information area and a mark file; the publisher stores the data to be transmitted into the shared memory and the metadata, including: the publisher stores the data to be transmitted into the pre-stored information area, and performs data decomposition on the data to be transmitted in the pre-stored information area; the publisher stores the service data of the data to be transmitted into the shared memory annular buffer area; the publisher stores basic information and data addresses of the data to be transmitted into the message metadata array, wherein the data addresses are storage addresses of the service data in the shared memory ring buffer; the publisher stores the topic name and the data type corresponding to the data to be transmitted into the mark file; subscribing the topic by a subscriber, and mapping metadata under the topic to a memory space of the subscriber; the subscriber takes out the data to be transmitted from the memory space through deserialization according to the metadata; the process of registering topic by the publisher further comprises: the publisher creates local inter-process communication to monitor connection requests; The subscriber subscribes to the topic, including: the subscriber monitors the mark file of the topic by using an inotify monitoring framework; And when the subscriber monitors that the publisher appears in the markup file, sending a connection request to the publisher, and adding a callback function provided by the publisher into a polling array.
  2. 2. The method as recited in claim 1, further comprising: and after the data to be transmitted is stored in the shared memory and the metadata, the publisher empties and releases the pre-stored information area.
  3. 3. The method of claim 1, wherein the process of retrieving the data to be transmitted in the memory space by deserialization by the subscriber based on the metadata comprises: the subscriber searches and determines a data address of service data of the data to be transmitted in a message metadata array of the metadata; the subscriber extracts from the shared memory annular buffer based on the data address of the data to be transmitted, and restores the service data of the data to be transmitted through deserialization; and the subscriber generates the data to be transmitted by calling the corresponding callback function.
  4. 4. The method of claim 1, wherein the process of retrieving the data to be transmitted in the memory space by deserialization by the subscriber based on the metadata, further comprises: and the subscriber confirms the validity of the data to be transmitted through the head and tail pointer position of the shared memory annular buffer zone and/or the head and tail pointer position of the message metadata array.
  5. 5. The method of claim 4, wherein the process of the subscriber confirming the validity of the data to be transmitted by the head and tail pointer location of the shared memory ring buffer comprises: The subscriber judges whether the memory data of the data to be transmitted is covered in the shared memory ring buffer area or not through the head pointer and the tail pointer of the shared memory ring buffer area; if the memory data is covered in the shared memory ring buffer, the subscriber confirms that the data to be transmitted is invalid; And if the memory data is not covered in the shared memory annular buffer area, the subscriber confirms that the data to be transmitted is valid.
  6. 6. The method of claim 4, wherein the process of the subscriber confirming the validity of the data to be transmitted by the head-to-tail pointer position of the message metadata array comprises: The subscriber judges whether the data address of the data to be transmitted is covered in the message metadata array or not according to the head pointer and the tail pointer of the message metadata array; if the data address is covered in the message metadata array, the subscriber confirms that the data to be transmitted is invalid; And if the data address is not covered in the message metadata array, the subscriber confirms that the data to be transmitted is valid.
  7. 7. A data transmission system comprising a publisher and a subscriber; the publisher registers topic by creating a shared memory and initializing metadata, and stores data to be transmitted into the shared memory and the metadata; The subscriber subscribes to the topic, metadata under the topic is mapped to a memory space of the subscriber, and the data to be transmitted is extracted from the memory space through deserialization according to the metadata; Wherein, the process of the publisher registering the topic by creating the shared memory and initializing the metadata includes: the publisher creates a shared memory and takes the shared memory as a shared memory ring buffer area under a topic, wherein the shared memory ring buffer area is provided with a head pointer and a tail pointer; initializing metadata by the publisher, wherein the metadata comprises a message metadata array with head and tail pointers, a pre-stored information area and a mark file; The process of storing the data to be transmitted into the shared memory and the metadata by the publisher comprises the following steps: the publisher stores the data to be transmitted into the pre-stored information area, and performs data decomposition on the data to be transmitted in the pre-stored information area; the publisher stores the service data of the data to be transmitted into the shared memory annular buffer area; the publisher stores basic information and data addresses of the data to be transmitted into the message metadata array, wherein the data addresses are storage addresses of the service data in the shared memory ring buffer; the publisher stores the topic name and the data type corresponding to the data to be transmitted into the mark file; the process of registering topic by the publisher further comprises: the publisher creates local inter-process communication to monitor connection requests; The subscriber subscribes to the topic, including: the subscriber monitors the mark file of the topic by using an inotify monitoring framework; And when the subscriber monitors that the publisher appears in the markup file, sending a connection request to the publisher, and adding a callback function provided by the publisher into a polling array.

Description

Data transmission method and system Technical Field The present application relates to the field of communications technologies, and in particular, to a data transmission method and system. Background With the increasing progress of society and the improvement of living standard of people, the importance of data is continuously enhanced, and the collection and transmission of data from mobile phones to intelligent home appliances to automobile industry are related to aspects of the progress of living technology. The ever-changing sensors for information collection, the ever-increasing networking devices create more data than ever before, and data transmission techniques need to keep pace. Taking the autopilot industry as an example, autopilot requirement equipment timely knows the surrounding environment, makes correct judgment according to the collected information, and then performs corresponding operation, so that a large number of sensors including a laser radar, a millimeter wave radar, an ultrasonic radar, a camera and the like are required to be installed in an autopilot system, and through flexible coordination among the sensors, timely response is performed to conduct predictive planning decision, and finally, automatic driving is realized, so that all passengers safely arrive at a destination. During this process, a large amount of sensor data transfer will occur. In this case, how to achieve high-performance data transmission between modules becomes a pain and difficulty that is currently in need of face and overcome. Disclosure of Invention In view of the above, the present application provides a data transmission method and system, which implement high performance data transmission between modules based on shared memory, lock-free data structure, zero copy, etc., and have the advantages of high throughput, low delay, no blocking, zero copy, no single point failure, etc. In order to achieve the above object, the following solutions have been proposed: a data transmission method, comprising: The publisher registers topic by creating a shared memory and initializing metadata; the publisher stores the data to be transmitted into the shared memory and the metadata; subscribing the topic by a subscriber, and mapping metadata under the topic to a memory space of the subscriber; And the subscriber takes out the data to be transmitted from the memory space through deserialization according to the metadata. Optionally, the process of registering topic by creating a shared memory and initializing metadata by the publisher includes: the publisher creates a shared memory and takes the shared memory as a shared memory ring buffer area under a topic, wherein the shared memory ring buffer area is provided with a head pointer and a tail pointer; The publisher initializes metadata including a message metadata array with head and tail pointers, a pre-stored information area, and a markup file. Optionally, the process of storing the data to be transmitted into the shared memory and the metadata by the publisher includes: The publisher stores the service data of the data to be transmitted into the pre-stored information area, and carries out data decomposition on the data to be transmitted in the pre-stored information area; the publisher stores the service data of the data to be transmitted into the shared memory annular buffer area; the publisher stores basic information and data addresses of the data to be transmitted into the message metadata array, wherein the data addresses are storage addresses of the service data in the shared memory ring buffer; and the publisher stores the topic name and the data type corresponding to the data to be transmitted into the mark file. Optionally, the method further comprises: and after the data to be transmitted is stored in the shared memory and the metadata, the publisher empties and releases the pre-stored information area. Optionally, the process of registering topic by the publisher further includes: the publisher creates local inter-process communication to monitor connection requests; The subscriber subscribes to the topic, including: the subscriber monitors the mark file of the topic by using an inotify monitoring framework; And when the subscriber monitors that the publisher appears in the markup file, sending a connection request to the publisher, and adding a callback function provided by the publisher into a polling array. Optionally, the process of retrieving the data to be transmitted in the memory space by deserializing by the subscriber according to the metadata includes: Searching and determining a data address of service data of the data to be transmitted in a message metadata array of the metadata; Extracting from the shared memory annular buffer area based on the data address of the data to be transmitted, and reducing the service data of the data to be transmitted through deserialization; And generating the data to be transmitted by calling the correspond