CN-122019588-A - Method, device, equipment and storage medium for processing tensor heterogeneous data

Abstract

The application provides a tensor heterogeneous data processing method, device, equipment and storage medium, wherein the method comprises the steps of obtaining a high-level query statement corresponding to data in a database table to be processed, analyzing the high-level query statement to obtain a logic execution plan corresponding to the high-level query statement, wherein the logic execution plan comprises a relational algebra operator, mapping the relational algebra operator into a tensor execution operator based on a query planner, and dynamically scheduling the tensor execution operator to a CPU or DPU (central processing unit) end for execution through a tensor execution engine.

Inventors

• Chen Gongye
• JIANG LI
• LIU FANGXIN

Assignees

• 上海交通大学

Dates

Publication Date

20260512

Application Date

20251224

Claims (10)

1. 1. A method of tensor heterogeneous data processing, the method comprising: acquiring a high-level query statement corresponding to data in a database table to be processed; analyzing the advanced query statement to obtain a logic execution plan corresponding to the advanced query statement, wherein the logic execution plan comprises a relational algebra operator; Mapping the relational algebra operator into a tensor execution operator based on a query planner; and the tensor execution operator is dynamically scheduled to a CPU or DPU end for execution through a tensor execution engine.
2. 2. The method of claim 1, wherein prior to retrieving the high-level query statement corresponding to the data in the database table to be processed, the method further comprises: determining a database table to be processed; and loading the data in the database table to a CPU end and a DPU end respectively.
3. 3. The method of claim 1, wherein the query planner is further responsible for synchronizing data across devices, ensuring data consistency and coordination between the CPU side and the DPU side, and ensuring proper execution of tensor operators on heterogeneous hardware.
4. 4. The method according to claim 1, wherein the tensor execution operator is dynamically scheduled to be executed by a CPU or a DPU end through a tensor execution engine, comprising: Marking operator categories for the tensor execution operators according to the execution characteristics of the tensor execution operators, wherein the operator categories comprise CPU friendly, DPU friendly and CPU-DPU mixed execution; and the tensor execution engine dynamically dispatches the tensor execution operator to a CPU or DPU end for execution based on the operator category.
5. 5. The method of claim 4, wherein the tensor execution engine dynamically schedules the tensor execution operator to a CPU or DPU side execution based on the operator class, comprising: If the operator class is CPU friendly, the tensor execution engine statically unloads tensor execution operators corresponding to the operator class to a CPU end for execution; if the operator class is executed by the DPU end, the tensor execution engine statically unloads tensor execution operators corresponding to the operator class to the DPU end for execution; And if the operator category is the CPU-DPU mixed execution, the tensor execution engine executes the tensor execution operator dynamic adjustment execution flow corresponding to the operator category at the CPU or DPU end.
6. 6. The method according to claim 1, wherein the method further comprises: mapping column-type data in a database table to be processed into a unified tensor structure; And the tensor execution operator executes corresponding data processing on the CPU or DPU end based on the tensor structure.
7. 7. The method of claim 1, wherein the relational algebra operator comprises at least one of Agg, join, filter.
8. 8. A tensor heterogeneous data processing apparatus, the apparatus comprising: the advanced query statement acquisition module is used for acquiring advanced query statements corresponding to the database tables to be processed; The logic execution plan determining module is used for analyzing the advanced query statement to obtain a logic execution plan corresponding to the advanced query statement, wherein the logic execution plan comprises a relational algebra operator; The mapping module is used for mapping the relational algebra operator into a tensor execution operator based on a query planner; and the execution module is used for dynamically scheduling the tensor execution operator to a CPU or DPU end for execution through a tensor execution engine.
9. 9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any of claims 1 to 7.
10. 10. An electronic device, the electronic device comprising: A memory storing a computer program; A processor in communication with the memory, the method of any of claims 1 to 7 being performed when the computer program is invoked.

Description

Method, device, equipment and storage medium for processing tensor heterogeneous data Technical Field The application belongs to the technical field of data processing, and relates to a tensor heterogeneous data processing method, device, equipment and storage medium. Background Currently, mainstream online analytical processing (OLAP) systems are commonly built on top of a general purpose computing paradigm with Central Processing Units (CPUs) as cores. Systems represented by MonetDB, vectorwise, clickHouse and DuckDB are highly convergent in terms of technology stack height, in that they generally employ a columnar storage model to optimize data compression rate and sequential I/O efficiency, and combine a vectorized execution engine with a multithreaded parallel technology to maximize the computational throughput of the CPU, thereby coping with the analysis query load of large-scale data. However, the performance bottlenecks of such systems are mainly due to memory bandwidth and data movement overhead, rather than the computing power of the CPU itself. Although modern CPUs have strong computational power, in the context of continuous expansion of data size and increasingly complex query loads, overall system performance is still limited by the data transfer rate between main memory and CPU cache. This phenomenon, the "memory wall" problem, reveals profoundly the fundamental imbalance caused by the fact that the rate of increase of the processor computation speed far exceeds the rate of increase of the memory access speed. Therefore, how to reduce data handling between the host and the memory and improve the overall system throughput is a technical problem to be solved. Disclosure of Invention The application provides a tensor heterogeneous data processing method, a tensor heterogeneous data processing device, tensor heterogeneous data processing equipment and a storage medium, which are used for dynamically executing data processing tasks at a CPU (Central processing Unit) or a DPU (data processing Unit) end and improving the data processing efficiency. The application provides a tensor heterogeneous data processing method, which comprises the steps of obtaining a high-level query statement corresponding to data in a database table to be processed, analyzing the high-level query statement to obtain a logic execution plan corresponding to the high-level query statement, wherein the logic execution plan comprises a relational algebra operator, mapping the relational algebra operator into a tensor execution operator based on a query planner, and dynamically scheduling the tensor execution operator to a CPU (Central processing Unit) or a DPU (data processing unit) end for execution through a tensor execution engine. In the application, a relational algebra operator is mapped into a tensor execution operator based on a query planner, a group of heterogeneous execution tensor operators suitable for a CPU and a PIM end are defined, the innovation ensures that data representation and calculation semantics are unified between the CPU and the PIM end, the technical innovation shields hardware details of the PIM end, simplifies programming complexity of upper-layer application, provides a solid foundation for subsequent SQL mapping and execution, and the tensor execution operator is dynamically scheduled to the CPU or DPU end for execution through scheduling of a tensor execution engine, so that high bandwidth and parallelism in the PIM can be fully utilized, and the capability of the CPU on global control and complex calculation is reserved. In one implementation manner of the first aspect, before acquiring the advanced query statement corresponding to the data in the database table to be processed, the method further includes determining the database table to be processed, and loading the data in the database table to the CPU side and the DPU side respectively. In an implementation manner of the first aspect, the query planner is further responsible for synchronizing data across devices, ensuring consistency and coordination of data between the CPU side and the DPU side, and ensuring correct execution of tensor operators on heterogeneous hardware. In one implementation manner of the first aspect, the tensor execution operator is dynamically scheduled to a CPU or a DPU end for execution through a tensor execution engine, and the tensor execution engine comprises marking operator categories for the tensor execution operator according to the execution characteristics of the tensor execution operator, wherein the operator categories comprise CPU friendly, DPU friendly and CPU-DPU mixed execution, and the tensor execution engine dynamically schedules the tensor execution operator to the CPU or the DPU end for execution based on the operator categories. In one implementation manner of the first aspect, the tensor execution engine dynamically schedules the tensor execution operator to a CPU or a DPU end for execution ba