Search

CN-122019270-A - Database log analysis recovery method, device, equipment, medium and program product

CN122019270ACN 122019270 ACN122019270 ACN 122019270ACN-122019270-A

Abstract

The application provides a database log analysis recovery method, a device, equipment, a medium and a program product, and relates to the field of big data. Generating a filtering rule according to blacklist configuration information configured by a user, carrying out matching judgment on to-be-processed transaction numbers in log data to be processed based on the filtering rule, marking the log data corresponding to the first transaction numbers as a filtering state and storing the first transaction numbers in a cache structure if the first transaction numbers in the to-be-processed transaction numbers are successfully matched in the filtering rule, analyzing the mode name and the table name of the log data corresponding to the second transaction numbers if the second transaction numbers in the to-be-processed transaction numbers are not successfully matched in the filtering rule and the second transaction numbers are not in a whitelist, and setting secondary filtering marks for the log data corresponding to the second transaction numbers if the mode name and the table name of the log data corresponding to the second transaction numbers are not analyzed.

Inventors

  • LI JIALUO
  • HE SU
  • WEI QIAN

Assignees

  • 中电科金仓(北京)科技股份有限公司

Dates

Publication Date
20260512
Application Date
20251229

Claims (10)

  1. 1. A log processing method, comprising: Generating a filtering rule according to blacklist configuration information configured by a user; The method comprises the steps of carrying out matching judgment on a to-be-processed transaction number in log data to be processed based on the filtering rule, wherein the to-be-processed transaction number is a number for identifying a transaction in the log data to be processed, and the to-be-processed transaction number supports a plurality of log entries in a correlated mode; if the first transaction number in the transaction numbers to be processed is successfully matched in the filtering rule, marking log data corresponding to the first transaction number as a filtering state, and storing the first transaction number into a cache structure, wherein the filtering state is used for indicating that a subsequent processing flow is skipped; If the second transaction number in the transaction numbers to be processed is not successfully matched in the filtering rule and the second transaction number is not in the white list, analyzing the mode name and the table name of log data corresponding to the second transaction number; and under the condition that the mode name and the table name of the log data corresponding to the second transaction number are not analyzed, setting a secondary filtering mark for the log data corresponding to the second transaction number.
  2. 2. The method according to claim 1, wherein the method further comprises: when the log data to be processed is subjected to secondary filtering, if the log data provided with the secondary filtering mark exists, acquiring the mode name and the table name of the log data provided with the secondary filtering mark; and marking the log data provided with the secondary filtering mark as the filtering state under the condition that the mode name and the table name of the log data provided with the secondary filtering mark are judged to be matched with the filtering rule.
  3. 3. The method according to claim 2, wherein the method further comprises: And adding the mode name and the table name of the log data provided with the secondary filtering mark into the white list under the condition that the mode name and the table name of the log data provided with the secondary filtering mark are not matched with the filtering rule.
  4. 4. A method according to any one of claims 1-3, wherein the blacklist configuration information includes a blacklist name and a transaction number, and the generating the filtering rule according to the blacklist configuration information configured by the user includes: And converting the blacklist table name and the transaction number into the matchable filtering rule, wherein the filtering rule comprises a transaction number matching rule and a table name matching rule.
  5. 5. The method according to claim 4, wherein the method further comprises: And dynamically adjusting the matching sequence of the filtering rules by combining the transaction type of the log data to be processed and an operation time stamp, wherein the transaction type comprises at least one of insertion, loading and deletion.
  6. 6. The method of claim 5, wherein said converting said blacklist table name and said transaction number into said filter rules that are matchable comprises: extracting a blacklist list and blacklist transaction number characteristics according to the blacklist list name and the transaction number; And converting the blacklist list name list and the blacklist transaction number characteristics into the matched filtering rules through a rule conversion engine.
  7. 7. A database log parsing and recovering apparatus, comprising: The generation module is used for generating a filtering rule according to blacklist configuration information configured by a user; The system comprises a filtering module, a matching module and a processing module, wherein the filtering module is used for filtering the log data to be processed according to the filtering rule, and the processing module is used for carrying out matching judgment on the transaction number to be processed in the log data to be processed, wherein the transaction number to be processed is used for identifying the number of the transaction in the log data to be processed and supports a plurality of log entries to be associated with the transaction number to be processed; The storage module is used for marking log data corresponding to a first transaction number as a filtering state and storing the first transaction number into a cache structure if the first transaction number in the transaction numbers to be processed is successfully matched in the filtering rule, wherein the filtering state is used for indicating that a subsequent processing flow is skipped; The analysis module is used for analyzing the mode name and the table name of the log data corresponding to the second transaction number if the second transaction number in the transaction numbers to be processed is not successfully matched in the filtering rule and the second transaction number is not in the white list; And the setting module is used for setting a secondary filtering mark for the log data corresponding to the second transaction number under the condition that the mode name and the table name of the log data corresponding to the second transaction number are not analyzed.
  8. 8. An electronic device is characterized by comprising a memory and a processor; The memory stores computer-executable instructions; The processor executing computer-executable instructions stored in the memory, causing the processor to perform the method of any one of claims 1-6.
  9. 9. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1-6.
  10. 10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-6.

Description

Database log analysis recovery method, device, equipment, medium and program product Technical Field The present application relates to the field of big data, and in particular, to a method, an apparatus, a device, a medium, and a program product for database log parsing recovery. Background With the acceleration of enterprise digital transformation, real-time data processing and synchronization demands are increasing. In critical business scenarios such as finance, logistics, e-commerce, internet of things, etc., users need to synchronize incremental data (e.g., transaction records, order status, device sensor data) in a database to downstream systems (e.g., data warehouse, real-time analysis platform, message queue, etc.) in a low-latency, highly reliable manner. In this process, a Write-ahead log (WAL) is used as a core mechanism for database record data changes, and its parsing efficiency and filtering capability directly affect the performance and stability of the entire data synchronization link. In the prior art, the method for analyzing and recovering the database log comprises the steps of firstly reading log data from a pre-written log file, performing preliminary filtration through a table physical identifier blacklist buffer, directly filtering if the table physical identifier corresponding to the log is in the buffer, continuing processing if the table physical identifier is not in the buffer, then performing first analysis on the log, storing the log into a transaction reordering buffer if the log is not a commit log after analysis, taking out the related log corresponding to the transaction from the transaction reordering buffer if the log is a commit log, matching the related log into a corresponding mode name and a table name according to the table physical identifier recorded in the log, performing regular judgment by combining with a configured blacklist filtering rule to complete secondary filtration, adding contents to be filtered into the table physical identifier blacklist buffer, and finally performing second analysis on the log to realize analysis and related logic processing of the pre-written log. However, in the prior art, only the white list parsing rule is supported, and the to-be-filtered table log is stored in the transaction reordering buffer first and then filtered in the secondary parsing stage, so that the buffer is excessively expanded and data is overflowed to a disk in a large number of newly built table scenes, and the problem of lower parsing performance exists. Disclosure of Invention The application provides a database log analysis recovery method, a device, equipment, a medium and a program product, which are used for solving the technical problem of lower analysis performance in the prior art. In a first aspect, the present application provides a method for recovering database log parsing, the method comprising: Generating a filtering rule according to blacklist configuration information configured by a user; The method comprises the steps of carrying out matching judgment on a to-be-processed transaction number in log data to be processed based on the filtering rule, wherein the to-be-processed transaction number is a number for identifying a transaction in the log data to be processed, and the to-be-processed transaction number supports a plurality of log entries in a correlated mode; if the first transaction number in the transaction numbers to be processed is successfully matched in the filtering rule, marking log data corresponding to the first transaction number as a filtering state, and storing the first transaction number into a cache structure, wherein the filtering state is used for indicating that a subsequent processing flow is skipped; If the second transaction number in the transaction numbers to be processed is not successfully matched in the filtering rule and the second transaction number is not in the white list, analyzing the mode name and the table name of log data corresponding to the second transaction number; and under the condition that the mode name and the table name of the log data corresponding to the second transaction number are not analyzed, setting a secondary filtering mark for the log data corresponding to the second transaction number. In one possible embodiment, the method further comprises: when the log data to be processed is subjected to secondary filtering, if the log data provided with the secondary filtering mark exists, acquiring the mode name and the table name of the log data provided with the secondary filtering mark; and marking the log data provided with the secondary filtering mark as the filtering state under the condition that the mode name and the table name of the log data provided with the secondary filtering mark are judged to be matched with the filtering rule. In one possible embodiment, the method further comprises: And adding the mode name and the table name of the log data provided with the secondary fil