CN-116633925-B - Data processing method and device

CN116633925BCN 116633925 BCN116633925 BCN 116633925BCN-116633925-B

Abstract

The application provides a data processing method and device. In the present application, identification information of the target rosbag data is acquired. The target rosbag data is located in the rosbag file stored in the cloud. And acquiring the offset position of the target rosbag data in the rosbag file according to the identification information of the target rosbag data and a first corresponding relation stored in advance, wherein the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file. And acquiring target rosbag data from the rosbag file stored in the cloud according to the offset position of the target rosbag data in the rosbag file. By the method and the device, time can be saved, and the efficiency of target rosbag data can be improved.

Inventors

LIU ZESHU
HUANG XIAOJING

Assignees

阿里云计算有限公司

Dates

Publication Date: 20260512
Application Date: 20230505

Claims (13)

1. A data processing method, applied to an electronic device, the method comprising: acquiring identification information of target rosbag data, wherein the target rosbag data is located in a rosbag file stored in a cloud; Acquiring the offset position of target rosbag data in a rosbag file according to the identification information of the target rosbag data and a first corresponding relation stored in advance, wherein the first corresponding relation comprises the corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file; Acquiring target rosbag data from rosbag files stored in a cloud according to the offset position; The target rosbag is multiple, and the offset positions of different target rosbag data in the rosbag file are different; The obtaining the target rosbag data from the rosbag files stored in the cloud according to the offset position includes: generating offset location intervals according to offset locations of the plurality of target rosbag data in the rosbag file respectively; generating a downloading request, wherein the downloading request at least carries an offset position interval; Sending a downloading request to the cloud end, so that the cloud end obtains data in an offset position interval from rosbag files stored in the cloud end according to the downloading request, and returns the data in the offset position interval to the electronic equipment; receiving data in an offset position interval returned by the cloud; And respectively acquiring the data of each target rosbag from the data in the offset position interval according to the offset position of the data of each target rosbag in the rosbag file.
2. The method according to claim 1, wherein the method further comprises: determining whether the target rosbag data is cached in the electronic device according to the offset position; And under the condition that the target rosbag data is not cached in the electronic equipment, executing the step of acquiring the target rosbag data from the rosbag file stored in the cloud according to the offset position.
3. The method according to claim 2, wherein the method further comprises: In the case where the target rosbag data has been cached in the electronic device, the target rosbag data cached in the electronic device is acquired.
4. A method according to claim 2 or 3, wherein said determining whether the target rosbag data has been cached in the electronic device based on said offset location comprises: searching rosbag data corresponding to the offset position in a second corresponding relation locally stored in the electronic equipment, wherein the second corresponding relation comprises a corresponding relation between the offset position of rosbag data in a rosbag file and rosbag data; Under the condition that rosbag data corresponding to the offset position is found in the second corresponding relation, determining that target rosbag data is cached in the electronic equipment; or if rosbag data corresponding to the offset position is not found in the second correspondence, determining that the target rosbag data is not cached in the electronic device.
5. The method of claim 1, wherein rosbag files stored in the cloud include a plurality of file blocks, wherein offset location intervals of different file blocks in rosbag files stored in the cloud are different; the generating an offset location interval according to the offset locations of the plurality of targets rosbag data in the rosbag file respectively includes: For any one file block, acquiring an offset position interval of the file block in rosbag files stored in a cloud; selecting an offset position located within an offset position interval of the file block from offset positions of a plurality of target rosbag data in rosbag files respectively; Screening the forefront offset position and the rearmost offset position among the offset positions in the offset position interval of the file block; an offset position section is generated from the forefront offset position and the rearmost offset position.
6. The method of claim 5, wherein determining whether the target rosbag data has been cached in the electronic device based on the offset location comprises: Determining an offset position interval in which the offset position of the target rosbag data in the rosbag file is located in the offset position interval in rosbag files stored in the cloud end respectively by different file blocks; determining whether a file block in which target rosbag data are located is cached in the electronic equipment according to the determined offset position interval; Determining that target rosbag data is cached in the electronic device in the case of a file block in which target rosbag data is cached in the electronic device; or in the case of a file block in which target rosbag data is not cached in the electronic device, determining that target rosbag data is not cached in the electronic device.
7. The method of claim 6, wherein determining whether the file block in which the target rosbag data is cached in the electronic device according to the determined offset location interval comprises: searching a file block corresponding to the determined offset position interval in a third corresponding relation locally stored in the electronic equipment, wherein the third corresponding relation comprises the corresponding relation between the offset position interval of the file block in rosbag files and the file block; Under the condition that the file block corresponding to the determined offset position interval is found in the third corresponding relation, determining the file block in which the cached target rosbag data in the electronic equipment is located; Or under the condition that the file block corresponding to the determined offset position interval is not found in the third corresponding relation, determining the file block where the uncached target rosbag data in the electronic equipment is located.
8. A method of data processing, the method comprising: in a scene that the automatic driving equipment uploads rosbag files to the cloud, acquiring the offset positions of identification information of each rosbag data and each rosbag data in rosbag files from rosbag files; Storing the offset positions of the identification information of each rosbag data and each rosbag data in the rosbag file respectively in a first corresponding relation, wherein the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file; generating an offset position interval according to offset positions of a plurality of target rosbag data in the rosbag file respectively, wherein the offset positions of different target rosbag data in the rosbag file are different; generating a downloading request, wherein the downloading request at least carries an offset position interval; Sending a downloading request to the cloud end, so that the cloud end obtains data in an offset position interval from rosbag files stored in the cloud end according to the downloading request, and returns the data in the offset position interval to the electronic equipment; receiving data in an offset position interval returned by the cloud; And respectively acquiring the data of each target rosbag from the data in the offset position interval according to the offset position of the data of each target rosbag in the rosbag file.
9. A data processing apparatus for application to an electronic device, the apparatus comprising: the first acquisition module is used for acquiring identification information of target rosbag data, wherein the target rosbag data is located in a rosbag file stored in a cloud; the second acquisition module is used for acquiring the offset position of the target rosbag data in the rosbag file according to the identification information of the target rosbag data and a first corresponding relation stored in advance, wherein the first corresponding relation comprises the corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file; The third obtaining module is configured to obtain target rosbag data from a rosbag file stored in the cloud according to the offset position; wherein the target rosbag data is a plurality of targets, and different targets rosbag data have different offset positions in the rosbag file, and the third obtaining module comprises: A first generation unit, configured to generate offset location intervals according to offset locations of the plurality of target rosbag data in the rosbag file, respectively; the second generating unit is used for generating a downloading request, and the downloading request at least carries an offset position interval; the sending unit is used for sending a downloading request to the cloud so that the cloud obtains data in an offset position interval from rosbag files stored in the cloud according to the downloading request and returns the data in the offset position interval to the electronic equipment; the receiving unit is used for receiving the data which are returned by the cloud and are positioned in the offset position interval; And an acquisition unit for acquiring each target rosbag data respectively from the data located in the offset position section according to the offset positions of each target rosbag data in the rosbag file respectively.
10. A data processing apparatus, the apparatus comprising: A fifth obtaining module, configured to obtain, in a scenario in which the autopilot device uploads the rosbag file to the cloud, offset positions of identification information of each rosbag data and each rosbag data in the rosbag file from the rosbag file; The storage module is used for storing the offset positions of the identification information of each rosbag data and each rosbag data in the rosbag file respectively in a first corresponding relation, wherein the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file; generating an offset position interval according to offset positions of a plurality of target rosbag data in the rosbag file respectively, wherein the offset positions of different target rosbag data in the rosbag file are different; generating a downloading request, wherein the downloading request at least carries an offset position interval; Sending a downloading request to the cloud end, so that the cloud end obtains data in an offset position interval from rosbag files stored in the cloud end according to the downloading request, and returns the data in the offset position interval to the electronic equipment; receiving data in an offset position interval returned by the cloud; And respectively acquiring the data of each target rosbag from the data in the offset position interval according to the offset position of the data of each target rosbag in the rosbag file.
11. An automatic driving apparatus, characterized in that the automatic driving apparatus comprises: and a memory associated with the one or more processors, the memory for storing program instructions that, when read for execution by the one or more processors, perform the following: in a scene that the automatic driving equipment uploads rosbag files to the cloud, acquiring the offset positions of identification information of each rosbag data and each rosbag data in rosbag files from rosbag files; Storing the offset positions of the identification information of each rosbag data and each rosbag data in the rosbag file respectively in a first corresponding relation, wherein the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file; generating an offset position interval according to offset positions of a plurality of target rosbag data in the rosbag file respectively, wherein the offset positions of different target rosbag data in the rosbag file are different; generating a downloading request, wherein the downloading request at least carries an offset position interval; Sending a downloading request to the cloud end, so that the cloud end obtains data in an offset position interval from rosbag files stored in the cloud end according to the downloading request, and returns the data in the offset position interval to the electronic equipment; receiving data in an offset position interval returned by the cloud; And respectively acquiring the data of each target rosbag from the data in the offset position interval according to the offset position of the data of each target rosbag in the rosbag file.
12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 8 when the program is executed by the processor.
13. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.

Description

Data processing method and device Technical Field The present application relates to the field of autopilot technology, and in particular, to a data processing method and apparatus. Background With the rapid development of technology, a ros operating system (robot operating system ING SYSTEM) is widely used in the fields of robots and autopilots. At present, an automatic driving platform based on a ros operation system can record information acquired by a sensor on automatic driving equipment into a rosbag-format file (rosbag file for short later) and upload the file to a cloud for storage. The sensor may include a camera, a radar, etc., the camera may collect image data, etc., the radar may include a laser radar, a millimeter wave radar, etc., the radar is used to collect radar data, the radar data may be used to measure distances or generate maps, etc. The rosbag file may include point cloud data, image data, log data, and annotation data, where the point cloud data may include radar data, and the log data may include an action performed by the autopilot device and a log associated with an algorithm related to the action, and the annotation data includes an object identified in the point cloud data or the image data, and the like. In some cases, a certain scene actually related to the autopilot device is played back according to actual situation needs, for example, a file rosbag can be downloaded from a cloud end by using a terminal in a laboratory, and data in the file rosbag is played back, so that an outdoor scene test autopilot function is simulated, a research and development personnel can quickly reproduce the actual scene based on offline data, and repeatable and low-cost analysis and debugging can be realized. However, in the actual use process, the occupied space of a single rosbag file is very large, for example, often reaches tens of GB or even hundreds of GB, and when playback is required, the rosbag file needs to be completely downloaded from the cloud to the terminal, then the complete rosbag file is subjected to data analysis and the like in the terminal, and then the analyzed data is played, so that playback of the scene actually related to the autopilot device is realized. However, the process of completely downloading rosbag files from the cloud to the local terminal consumes long time, is low in efficiency, consumes more network resources, and the like. Disclosure of Invention The application discloses a data processing method and device. In a first aspect, the application discloses a data processing method applied to electronic equipment, which comprises the steps of obtaining identification information of target rosbag data, wherein target rosbag data is located in a rosbag file stored in a cloud, obtaining an offset position of target rosbag data in a rosbag file according to the identification information of target rosbag data and a first corresponding relation stored in advance, wherein the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file, and obtaining target rosbag data from a rosbag file stored in the cloud according to the offset position. In a second aspect, the application discloses a data processing method, which comprises the steps of acquiring offset positions of identification information of each piece of rosbag data and each piece of rosbag data in rosbag files respectively from rosbag files in a scene that automatic driving equipment uploads rosbag files to a cloud, and storing the offset positions of identification information of each piece of rosbag data and each piece of rosbag data in rosbag files respectively in a first corresponding relation, wherein the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in rosbag files and the offset positions of rosbag data in rosbag files. In a third aspect, the application discloses a data processing device, which is applied to electronic equipment, and comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring identification information of target rosbag data, the target rosbag data is located in a rosbag file stored in a cloud, the second acquisition module is used for acquiring an offset position of target rosbag data in a rosbag file according to the identification information of the target rosbag data and a first corresponding relation stored in advance, the first corresponding relation comprises a corresponding relation between the identification information of rosbag data in the rosbag file and the offset position of rosbag data in the rosbag file, and the third acquisition module is used for acquiring target rosbag data from a rosbag file stored in the cloud according to the offset position. In a fourth aspect, the