Search

CN-121979896-A - Log playback method, device, computer equipment, storage medium and program product

CN121979896ACN 121979896 ACN121979896 ACN 121979896ACN-121979896-A

Abstract

The invention discloses a log playback method, a log playback device, computer equipment, a storage medium and a program product. The method comprises the steps of analyzing a log file to be processed, extracting current target transaction characteristics in real time, predicting lock information in the current target transaction and conflict probability between the current target transaction and a current active transaction by adopting a trained lock prediction model according to the current target transaction characteristics, constructing a lock thermodynamic diagram according to the lock information and the conflict probability, dividing a non-conflict area and a conflict area, merging the transactions in the non-conflict area, distributing the transactions to a plurality of space-time fragments to concurrently execute playback tasks, dividing the transactions in the conflict area to a plurality of time window intervals, and distributing the transactions in the target time window interval to the plurality of space-time fragments to concurrently execute the playback tasks if no conflict exists in the target time window interval. The technical scheme provided by the invention can improve the log playback speed, reduce the resource consumption and effectively solve the problem of concurrent conflict.

Inventors

  • LIU HAINAN

Assignees

  • 金篆信科有限责任公司

Dates

Publication Date
20260505
Application Date
20260116

Claims (10)

  1. 1. A log playback method, comprising: Analyzing the log file to be processed, and extracting in real time to obtain the characteristics of the current target transaction; According to the characteristics of the current target transaction, predicting the lock information in the current target transaction and the collision probability between the current target transaction and the current active transaction by adopting a trained lock prediction model; constructing a lock thermodynamic diagram according to the lock information and the conflict probability, and dividing a non-conflict area and a conflict area; merging the transactions in the non-conflict area, and distributing the transactions to a plurality of space-time slices so as to concurrently execute playback tasks; Dividing the transaction in the conflict zone into a plurality of time window intervals, and if no conflict exists in the target time window interval, distributing the transaction in the target time window interval to a plurality of time-space fragments so as to concurrently execute the playback task.
  2. 2. The method of claim 1, further comprising, prior to said predicting, using a trained lock prediction model, lock information in a current target transaction and a probability of collision with a current active transaction based on the current target transaction characteristics: in a production or pressure measurement environment, starting a full SQL audit log and a lock waiting log; Correlating all SQL sentences in the same transaction with precompiled parameters, execution time stamps and commit/rollback results; extracting lock features actually held and waited by the transaction from the beginning of the transaction to the commit, and forming a transaction-lock relation pair; and training the lock prediction model according to the transaction-lock relation pair generated training sample data.
  3. 3. The method of claim 1, wherein the current target transaction characteristics include static characteristics including SQL statement fingerprints, display lock hints, and operation types and objects, dynamic context characteristics including input parameter values/ranges and current database states including data distribution, index cases, and system loads, and historical behavior characteristics including historical lock matrices and historical conflict records.
  4. 4. The log playback method as recited in claim 1, wherein the method further comprises: monitoring lock conflicts actually occurring in real time; If the two transactions are detected to conflict, one of the transactions is selected to roll back and replay; If the number of times of conflict of the target space-time fragments exceeds a preset threshold, splitting the target space-time fragments based on the time-space distribution of the conflict, and/or adjusting the time window interval.
  5. 5. The log playback method as recited in claim 4, wherein the method further comprises: if the conflict is detected, new training sample data is generated according to the conflict transaction, and incremental training is carried out on the lock prediction model.
  6. 6. The log playback method as recited in claim 1, wherein parsing the log file to be processed comprises: identifying the log format of the log file to be processed according to a predefined grammar rule and a semantic template; splitting the log file to be processed into a plurality of subfiles according to the identification result; And analyzing the plurality of subfiles in parallel.
  7. 7. A log playback apparatus, comprising: the feature extraction module is used for analyzing the log file to be processed and extracting the current target transaction feature in real time; The lock information and conflict prediction module is used for predicting the lock information in the current target transaction and the conflict probability between the current target transaction and the current active transaction by adopting a trained lock prediction model according to the characteristics of the current target transaction; the lock thermodynamic diagram construction module is used for constructing a lock thermodynamic diagram according to the lock information and the conflict probability and dividing a non-conflict area and a conflict area; The non-conflict area transaction playback module is used for merging the transactions in the non-conflict area and distributing the transactions to a plurality of time-space fragments so as to concurrently execute playback tasks; And the conflict zone transaction playback module is used for dividing the transaction in the conflict zone into a plurality of time window intervals, and if no conflict exists in the target time window interval, distributing the transaction in the target time window interval to a plurality of time-space fragments so as to concurrently execute playback tasks.
  8. 8. A computer device, comprising: One or more processors; A memory for storing one or more programs; When executed by the one or more processors, cause the one or more processors to implement the log playback method of any one of claims 1-6.
  9. 9. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the log playback method of any one of claims 1-6.
  10. 10. A computer program product comprising a computer program which, when executed by a processor, implements the log playback method of any one of claims 1-6.

Description

Log playback method, device, computer equipment, storage medium and program product Technical Field The embodiment of the invention relates to the technical field of databases, in particular to a log playback method, a log playback device, computer equipment, a storage medium and a program product. Background In database systems, binary logs (Binlog) are used to record database modification operations, such as data insertion, update, and deletion, and Binlog plays a vital role in data recovery, data replication, and data synchronization. However, with the increasing data volume and the increasing business complexity, the conventional Binlog playback method faces many challenges in terms of performance. Firstly, when a large number of Binlogs are processed, the traditional single-thread playback mode is low in efficiency, so that the data recovery or synchronization time is too long, and the continuity and usability of the service are seriously affected. As in some data-intensive e-commerce databases, a complete Binlog playback may take hours or even days during which the service may be in an unavailable or data inconsistent state. Secondly, the long-time Binlog playback process occupies a large amount of system resources including CPU, memory, disk I/O, etc., which not only affects the performance of the database system itself, but may also negatively affect other applications running on the same server. As in some resource-limited cloud server environments, the Binlog playback process may cause excessive system load, and other business applications may respond slowly or even fail to work properly. Furthermore, when multi-thread parallel playback is attempted, due to the front-back dependency relationship between Binlog events, for example, a plurality of operations in a transaction need to be executed in sequence, the problem of concurrency conflict is easy to occur. Existing concurrency control mechanisms often require the introduction of a large number of lock mechanisms when dealing with these complex dependencies, which in turn further reduces the concurrency performance of playback. As in some high concurrency writing database scenarios, lock contention frequently occurs with multi-line Cheng Huifang Binlog, resulting in playback speeds that are instead less than single-threaded. In summary, the existing Binlog playback technology has slow playback speed, high resource consumption, and difficult concurrent conflict processing, and cannot meet the increasing data processing requirements. Disclosure of Invention The embodiment of the invention provides a log playback method, a log playback device, computer equipment, a storage medium and a program product, which are used for improving Binlog playback speed, reducing resource consumption and effectively solving the problem of concurrency conflict. In a first aspect, an embodiment of the present invention provides a log playback method, where the method includes: Analyzing the log file to be processed, and extracting in real time to obtain the characteristics of the current target transaction; According to the characteristics of the current target transaction, predicting the lock information in the current target transaction and the collision probability between the current target transaction and the current active transaction by adopting a trained lock prediction model; constructing a lock thermodynamic diagram according to the lock information and the conflict probability, and dividing a non-conflict area and a conflict area; merging the transactions in the non-conflict area, and distributing the transactions to a plurality of space-time slices so as to concurrently execute playback tasks; Dividing the transaction in the conflict zone into a plurality of time window intervals, and if no conflict exists in the target time window interval, distributing the transaction in the target time window interval to a plurality of time-space fragments so as to concurrently execute the playback task. Optionally, before predicting, according to the current target transaction characteristic, the lock information in the current target transaction and the collision probability with the current active transaction by using the trained lock prediction model, the method further includes: in a production or pressure measurement environment, starting a full SQL audit log and a lock waiting log; Correlating all SQL sentences in the same transaction with precompiled parameters, execution time stamps and commit/rollback results; extracting lock features actually held and waited by the transaction from the beginning of the transaction to the commit, and forming a transaction-lock relation pair; and training the lock prediction model according to the transaction-lock relation pair generated training sample data. Optionally, the current target transaction characteristic comprises a static characteristic, a dynamic context characteristic and a historical behavior characteristic, wher