CN-116303525-B - Data processing method, storage medium and device for database

CN116303525BCN 116303525 BCN116303525 BCN 116303525BCN-116303525-B

Abstract

The invention provides a data processing method, a storage medium and equipment of a database, wherein the data processing method comprises the steps of analyzing a transaction to obtain an analysis result, screening to obtain a transaction to be processed according to the analysis result, judging whether the transaction to be processed is a target transaction according to a statement in the transaction to be processed, adding a preset processing mark to a preset position of the target transaction, packaging, and sending the packaged target transaction to a target end. The database data processing method of the invention marks the transaction attribute in advance, so that the transaction containing the preset processing mark is identified in advance in the process of source end analysis, and marked on the transaction attribute, when the transaction is transmitted to the target end, the target end can select a relatively fast warehousing mode according to the attribute, thereby accelerating the data processing efficiency.

Inventors

WANG KAILONG
LENG JIANQUAN

Assignees

北京人大金仓信息技术股份有限公司

Dates

Publication Date: 20260505
Application Date: 20221208

Claims (8)

1. A data processing method of a database, comprising: Analyzing the transaction to obtain an analysis result; Screening according to the analysis result to obtain a transaction to be processed; judging whether the transaction to be processed is a target transaction according to the statement in the transaction to be processed; Adding a preset processing mark to the preset position of the target transaction, and packaging; sending the packed target transaction to a target end; the step of screening to obtain the transaction to be processed according to the analysis result comprises the following steps: judging whether the analysis result contains a setting mark for submitting or not; If yes, determining the transaction as a transaction to be processed; The transaction to be processed comprises a plurality of sentences, and the step of judging whether the transaction to be processed is a target transaction according to the sentences in the transaction to be processed comprises the following steps: Adding a correct mark to the transaction to be processed; sequentially determining the statement types of the statements in the transaction to be processed from head to tail, and Updating the mark of the transaction to be processed according to the statement type; And when updating to the tail part of the transaction to be processed, determining the transaction to be processed as a target transaction if the mark is still the correct mark.
2. The data processing method according to claim 1, wherein the step of updating the tag of the transaction to be processed according to the statement type comprises: If the statement is a data manipulation statement, judging whether the data manipulation statement is an insertion statement or not; If yes, judging whether the table name related to the data manipulation statement is the same as the table name in the previous data manipulation statement; if not, carrying out error marking on the transaction to be processed; If yes, the transaction to be processed is marked correctly.
3. The data processing method according to claim 2, wherein after the step of judging the data manipulation sentence as an insert sentence, further comprising: and in the case that the data manipulation statement is not an insert statement, performing the error marking on the transaction to be processed.
4. A data processing method according to claim 3, wherein the step of updating the tag of the transaction to be processed according to statement type comprises: And if the statement is a data definition statement, carrying out the error marking on the transaction to be processed.
5. The data processing method according to claim 1, wherein, The preset position is the head position of the target transaction.
6. The data processing method according to claim 1, wherein, The preset processing marks are batch processing marks.
7. A machine-readable storage medium having stored thereon a machine-executable program which, when executed by a processor, implements a data processing method of a database according to any of claims 1 to 6.
8. A computer device comprising a memory, a processor and a machine executable program stored on the memory and running on the processor, and the processor implementing a data processing method of a database according to any one of claims 1 to 6 when executing the machine executable program.

Description

Data processing method, storage medium and device for database Technical Field The present invention relates to the field of databases, and in particular, to a method for processing a database, a storage medium, and a device. Background The method comprises the steps of carrying out real-time data synchronization by utilizing data synchronization software, dividing the real-time data synchronization into three stages, carrying out initialization loading of stock data in the first stage to obtain a data synchronization basic point, carrying out incremental data synchronization by taking the synchronization basic point established by the initialization data loading as a reference in the second stage, and carrying out comparison and verification on source data and target data of the data synchronization in a regular period to confirm that the data is not lost in the data synchronization process. Wherein the second stage and the third stage are in a state of long-time parallelism. In the second stage, incremental data synchronization is performed by analyzing a database LOG (REDO LOG) to obtain incremental data, so that real-time synchronization of data is a common real-time data replication technology. The technology obtains the addition, deletion and modification changes of data by analyzing an ONLINE LOG (ONLINE LOG) or an archive LOG (ARCHIVE LOG) of a source database, converts the changes into a specific message format in synchronous software by taking a transaction as a unit, sends the changes to target-end data synchronous software through a private transmission protocol of the data synchronous software, and finally the target-end synchronous software restores the obtained transaction LOG into SQL sentences supported by the target database and executes the SQL sentences on the target-end database so as to realize real-time synchronization of the data, thereby maintaining the data consistency of the source-end and the target database. For some analytical database systems, operations like INSERT INTO SELECT or CREATE TABLE AS typically occur, which are characterized by oversized transactions that occur in a short period of time and are all new data. The data synchronization software is a general software, and needs to process INSERT, UPDATE, DELETE and DDL operations simultaneously, and when the target end enters the library, the attribute (INSERT, UPDATE, DELETE) of the next statement cannot be predicted, so that the next statement is always processed one by one, and if oversized transactions generated by operations such AS INSERT INTO SELECT or CREATE TABLE AS are encountered, the operations are treated AS independent INSERT operations one by one even if the operations are analyzed into one transaction. For tens of millions of levels of tables, the copying using INSERT INTO SELECT may take only a few minutes at the source and may take several hours to put in storage at the destination. Disclosure of Invention An object of the present invention is to provide a data processing method, a storage medium, and an apparatus of a database capable of solving any of the above problems. A further object of the present invention is to improve data processing efficiency. In particular, the present invention provides a data processing method of a database, the data processing method comprising: Analyzing the transaction to obtain an analysis result; screening according to the analysis result to obtain a transaction to be processed; judging whether the transaction to be processed is a target transaction according to the statement in the transaction to be processed; adding a preset processing mark to a preset position of a target transaction, and packaging; And sending the packed target transaction to the target end. Further, the step of screening to obtain the transaction to be processed according to the analysis result comprises the following steps: judging whether the analysis result contains a setting mark for submitting; if yes, determining the transaction as a to-be-processed transaction. Further, the transaction to be processed includes a plurality of sentences, and the step of judging whether the transaction to be processed is a target transaction according to the sentences in the transaction to be processed includes: adding a correct mark to the transaction to be processed; Sequentially determining the statement types of the statements in the transaction to be processed from head to tail, and Updating the mark of the transaction to be processed according to the statement type; And determining the transaction to be processed as a target transaction under the condition that the tail mark of the transaction to be processed is still the correct mark. Further, the step of updating the tag of the transaction to be processed according to the statement type includes: if the statement is a data manipulation statement, judging whether the data manipulation statement is an insertion statement or not; if yes, judging wheth