CN-122019647-A - Transaction data processing method, apparatus, device, storage medium and program product

Abstract

The embodiment of the application provides a transaction data processing method, a device, equipment, a storage medium and a program product, and relates to the field of database management. The method comprises the steps of obtaining first transaction change data corresponding to a first transaction, extracting a plurality of filtering functions of a plurality of operators corresponding to a first workflow, wherein the first workflow is used for processing the first transaction change data, constructing a corresponding ordered function chain according to the filtering functions based on data flow topology information of the first workflow, and sequentially processing the first transaction change data according to the ordered function chain. The method provided by the application improves the processing efficiency of the transaction data.

Inventors

• HU RUI
• HE SU
• WEI QIAN

Assignees

• 中电科金仓(北京)科技股份有限公司

Dates

Publication Date

20260512

Application Date

20251229

Claims (10)

1. 1. A transaction data processing method, the method comprising: acquiring first transaction change data corresponding to a first transaction; extracting a plurality of filter functions of a plurality of operators corresponding to a first workflow, wherein the first workflow is used for processing the first transaction change data; Constructing a corresponding ordered function chain according to the plurality of filtering functions based on the data stream topology information of the first workflow; And sequentially processing the first transaction change data according to the ordered function chain.
2. 2. The method of claim 1, wherein the constructing a corresponding ordered function chain from the plurality of filter functions based on the data flow topology information of the first workflow comprises: determining a topological order of the plurality of filter functions according to data flow topological information of the plurality of operators in the first workflow; Based on the topological order of the plurality of filtering functions, constructing a corresponding ordered function chain according to the plurality of filtering functions.
3. 3. The method of claim 1, wherein extracting a plurality of filter functions for a plurality of operators corresponding to the first workflow comprises: extracting a plurality of filter functions of a plurality of operators corresponding to the first workflow through a reflection mechanism, or, And extracting a plurality of filter functions of a plurality of operators corresponding to the first workflow through the operator registry.
4. 4. The method according to claim 1, wherein the method further comprises: Acquiring the data stream topology information updated by the first workflow; reconstructing a corresponding ordered function chain based on the updated data stream topology information of the first workflow.
5. 5. The method of claim 1, wherein the ordered function chain comprises a plurality of function sub-chains that match conditional logic information of the plurality of operators.
6. 6. The method of claim 5, wherein sequentially processing the first transaction change data according to the ordered function chain comprises: determining an objective function sub-chain in a plurality of function sub-chains of the ordered function chain according to the data characteristics corresponding to the first transaction change data; and sequentially processing the first transaction change data according to the target function sub-chain.
7. 7. The method of claim 1, wherein the obtaining the first transaction change data corresponding to the first transaction comprises: Determining a corresponding first transaction identifier and a plurality of pieces of SQL operation data according to the first data change event information of the first transaction; And determining the first transaction change data according to the plurality of SQL operation data.
8. 8. The method of claim 7, wherein after said sequentially processing said first transaction change data according to said ordered chain of functions, said method further comprises: and generating the processed first data change event information according to the first transaction change data and the first transaction identifier which are sequentially processed by the ordered function chain.
9. 9. A transaction data processing device, the transaction data processing device comprising: The acquisition module is used for acquiring first transaction change data corresponding to the first transaction; The processing module is used for extracting a plurality of filtering functions of a plurality of operators corresponding to a first workflow, wherein the first workflow is used for processing the first transaction change data, constructing a corresponding ordered function chain according to the plurality of filtering functions based on the topology information of the data flow of the first workflow, and sequentially processing the first transaction change data according to the ordered function chain.
10. 10. An electronic device comprising a processor and a memory communicatively coupled to the processor; The memory stores computer-executable instructions; The processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1 to 8.

Description

Transaction data processing method, apparatus, device, storage medium and program product Technical Field The present application relates to the field of database management, and in particular, to a transaction data processing method, apparatus, device, storage medium, and program product. Background In the data synchronization process, the source database encapsulates transaction-level data changes (such as INSERT, UPDATE, DELETE operations) into Event data structures containing multiple structured query language (structured query language, SQL) operations by parsing the online log or archive log, and transmits the Event data structures to the target database through a proprietary protocol. The target end needs to analyze, filter and convert the data in the Event item by item, finally generates SQL sentences supported by the target database and executes the SQL sentences so as to realize the data consistency of the source end and the target end. In the incremental synchronization phase, the transaction log of the source database typically contains compound Event of multiple SQL operations, e.g., one transaction contains associated update operations to multiple tables. However, in the above implementation manner, each operator (such as data format conversion, field filtering, condition filtering, etc.) in the extract-transform-load (ETL) process needs to independently traverse all data records in the Event, and the same Event may be scanned multiple times, resulting in lower processing efficiency for the transaction change Event. Disclosure of Invention The application provides a transaction data processing method, a device, equipment, a storage medium and a program product, which are used for solving the technical problem of poor management capability of transaction data. The application provides a transaction data processing method, which comprises the steps of obtaining first transaction change data corresponding to a first transaction, extracting a plurality of filter functions of a plurality of operators corresponding to a first workflow, wherein the first workflow is used for processing the first transaction change data, constructing a corresponding ordered function chain according to the plurality of filter functions based on data flow topology information of the first workflow, and sequentially processing the first transaction change data according to the ordered function chain. In one possible implementation manner of the first aspect, the constructing a corresponding ordered function chain according to the plurality of filtering functions based on the data flow topology information of the first workflow includes determining a topology order of the plurality of filtering functions according to the data flow topology information of the plurality of operators in the first workflow, and constructing a corresponding ordered function chain according to the plurality of filtering functions based on the topology order of the plurality of filtering functions. In one possible implementation manner of the first aspect, extracting the plurality of filter functions of the plurality of operators corresponding to the first workflow includes extracting the plurality of filter functions of the plurality of operators corresponding to the first workflow through a reflection mechanism, or extracting the plurality of filter functions of the plurality of operators corresponding to the first workflow through an operator registry. In a possible implementation manner of the first aspect, the method further includes obtaining updated data flow topology information of the first workflow, and reconstructing a corresponding ordered function chain based on the updated data flow topology information of the first workflow. In a possible implementation manner of the first aspect, the ordered function chain includes a plurality of function sub-chains, and the plurality of sub-chains are matched with conditional logic information of a plurality of operators. In a possible implementation manner of the first aspect, the processing the first transaction change data sequentially according to the ordered function chain includes determining an objective function sub-chain of a plurality of function sub-chains of the ordered function chain according to a data feature corresponding to the first transaction change data, and processing the first transaction change data sequentially according to the objective function sub-chain. In one possible implementation manner of the first aspect, acquiring the first transaction change data corresponding to the first transaction includes determining a corresponding first transaction identifier and a plurality of pieces of SQL operation data according to the first data change event information of the first transaction, and determining the first transaction change data according to the plurality of pieces of SQL operation data. In a possible implementation manner of the first aspect, after sequentially pr