Search

CN-121979750-A - Log data processing method, device, equipment, readable storage medium and program product

CN121979750ACN 121979750 ACN121979750 ACN 121979750ACN-121979750-A

Abstract

The present application relates to a log data processing method, apparatus, computer device, computer readable storage medium and computer program product. The method comprises the steps of collecting an original log from a log stream of a database, obtaining a target log according to the original log and metadata of the original log, determining log stream characteristics of the log stream, wherein the log stream characteristics comprise log generation rate, file size and time window of the log in the log stream, determining a slicing triggering condition according to the log stream characteristics, slicing the target log to obtain log blocks under the condition that the target log meets the slicing triggering condition, and carrying out parallel processing on the log blocks to obtain log processing results for the database. By adopting the method, the processing efficiency of the log data can be improved.

Inventors

  • WU LIANGLIANG

Assignees

  • 天翼云科技有限公司

Dates

Publication Date
20260505
Application Date
20251222

Claims (15)

  1. 1. A method of log data processing, the method comprising: collecting an original log from a log stream of a database, and obtaining a target log according to the original log and metadata of the original log; Determining log stream characteristics of the log stream, wherein the log stream characteristics comprise log generation rate, file size and time window of a log in the log stream; determining a slicing triggering condition according to the log stream characteristics, and slicing the target log under the condition that the target log meets the slicing triggering condition to obtain log blocks; and carrying out parallel processing on each log block to obtain a log processing result aiming at the database.
  2. 2. The method according to claim 1, wherein determining a slicing trigger condition according to the log stream feature, and splitting the target log to obtain log blocks when the target log meets the slicing trigger condition, includes: determining a predicted log generation rate according to the obtained historical flow characteristics of the log flow and the log flow characteristics; obtaining at least one fragmentation triggering condition based on the predicted log generation rate; And under the condition that the target log meets the at least one fragmentation triggering condition, cutting the target log to obtain log blocks.
  3. 3. The method of claim 2, wherein deriving at least one shard trigger condition based on the predicted log generation rate comprises: Based on the prediction log generation rate, respectively determining dynamic thresholds, wherein the dynamic thresholds comprise at least one of event number thresholds, volume thresholds or time thresholds; and constructing at least one fragmentation triggering condition according to the dynamic threshold.
  4. 4. The method according to claim 2, wherein, in the case that the target log meets the at least one slicing trigger condition, performing slicing on the target log to obtain log blocks, including: And under the condition that the target log meets the target trigger condition in the at least one fragmentation trigger condition, segmenting the target log according to the target trigger condition to obtain each log block.
  5. 5. The method according to claim 1, wherein the parallel processing for each log block to obtain a log processing result for the database includes: For each log block in the log blocks, loading the log block to a user memory space in a memory mapping mode; Carrying out structuring processing on the log blocks in the user memory space to obtain structured log blocks; storing the structured log blocks to a target storage system to obtain block processing results of the log blocks; and obtaining the log processing result of the database according to the respective block processing result of each log block.
  6. 6. The method of claim 5, wherein the structuring the log block in the user memory space to obtain a structured log block comprises: according to a preset log format, carrying out structural analysis on the log block in the user memory space to obtain a log block in a standardized format; And under the condition that the log blocks in the standardized format meet the reservation condition, obtaining the structured log blocks according to the log blocks in the standardized format.
  7. 7. The method according to claim 1, wherein the collecting an original log from a log stream of a database, and obtaining a target log according to the original log and metadata of the original log, comprises: collecting an original log from a log stream of a database based on a database agent; Determining metadata of the original log, wherein the metadata comprises identification, attribute information and verification information of the original log; And obtaining a target log according to the original log and the metadata.
  8. 8. The method of claim 7, wherein the determining metadata of the original log comprises: Generating an identifier of the original log according to the timestamp, the machine identifier and the serial number of the original log; obtaining attribute information of the original log according to the content of at least one structured field in the log content of the original log; obtaining verification information of the original log according to the log content of the original log; And obtaining metadata of the original log based on the identification, the attribute information and the verification information.
  9. 9. The method of claim 8, wherein the at least one structured field comprises at least one of operation user information, operation type, execution time, and scope of influence information.
  10. 10. The method of claim 1, wherein the determining log flow characteristics of the log flow comprises: Determining a time window; According to the time window, statistics is carried out on the logs in the log stream, so that the log generation rate and the file size of the logs in the log stream are obtained; and obtaining the log stream characteristics of the log stream according to the time window, the log generation rate and the file size.
  11. 11. The method according to any one of claims 1 to 10, further comprising: generating respective metadata records of the log blocks; under the condition that the state snapshot triggering condition is met, generating a global state snapshot; and under the condition that the data recovery triggering condition is met, carrying out data recovery according to the global state snapshot and the metadata records of each log block.
  12. 12. A log data processing apparatus, the apparatus comprising: The log acquisition module is used for acquiring an original log from a log stream of a database and obtaining a target log according to the original log and metadata of the original log; The system comprises a log stream feature determining module, a log stream feature determining module and a log stream processing module, wherein the log stream feature determining module is used for determining log stream features of the log stream, and the log stream features comprise log generation rate, file size and time window of a log in the log stream; The log segmentation module is used for determining a slicing triggering condition according to the log stream characteristics, and segmenting the target log under the condition that the target log meets the slicing triggering condition to obtain each log block; And the parallel processing module is used for carrying out parallel processing on each log block to obtain a log processing result aiming at the database.
  13. 13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 11 when the computer program is executed.
  14. 14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 11.
  15. 15. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 11.

Description

Log data processing method, device, equipment, readable storage medium and program product Technical Field The present application relates to the field of database technologies, and in particular, to a log data processing method, apparatus, computer device, computer readable storage medium, and computer program product. Background Along with the development of database technology, various database systems have significantly advanced in the aspects of data storage, management and application, log data is used as key information for recording the operation behaviors and states of the database, and the processing efficiency is critical to the optimization of system performance, fault tracing and security audit. At present, the processing of the database log data can be fluctuated due to factors such as business peaks, valleys and the like, and the processing efficiency of the database log data is low when the database log data is faced with high concurrency and massive log scenes. Disclosure of Invention In view of the foregoing, it is desirable to provide a log data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve log data processing efficiency. In a first aspect, the present application provides a log data processing method, including: Collecting an original log from a log stream of a database, and obtaining a target log according to the original log and metadata of the original log; determining the log stream characteristics of the log stream, wherein the log stream characteristics comprise the log generation rate, the file size and the time window of the log in the log stream; determining a slicing triggering condition according to the log stream characteristics, and slicing the target log under the condition that the target log meets the slicing triggering condition to obtain each log block; And carrying out parallel processing on each log block to obtain a log processing result for the database. In a second aspect, the present application also provides a log data processing apparatus, including: The log acquisition module is used for acquiring an original log from a log stream of the database and obtaining a target log according to the original log and metadata of the original log; The system comprises a log stream feature determining module, a log stream feature determining module and a log stream processing module, wherein the log stream feature determining module is used for determining the log stream feature of a log stream, and the log stream feature comprises the log generation rate, the file size and the time window of a log in the log stream; the log segmentation module is used for determining a slicing triggering condition according to the log stream characteristics, and segmenting the target log to obtain each log block under the condition that the target log meets the slicing triggering condition; And the parallel processing module is used for carrying out parallel processing on each log block to obtain a log processing result aiming at the database. In a third aspect, the present application also provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the method provided in the first aspect above when the computer program is executed by the processor. In a fourth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method provided in the first aspect above. In a fifth aspect, the application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method provided in the first aspect above. The method, the device, the computer equipment, the computer readable storage medium and the computer program product for processing the log data acquire the original log from the log stream of the database, acquire the target log according to the original log and the metadata of the original log, determine the slicing triggering condition according to the log stream characteristics of the log stream, wherein the log stream characteristics comprise the log generation rate, the file size and the time window of the log in the log stream, and cut the target log under the condition that the target log meets the slicing triggering condition, and perform parallel processing on each log block obtained by cutting to acquire the log processing result for the database. For a target log obtained from a log stream of a database, determining a slicing triggering condition according to log stream characteristics including the log generation rate, the file size and the time window of the log in the log stream, and performing parallel processing after dynamically slicing the target log according to the slicing triggering condition, so that system resources can be fully utilized base