Search

CN-122019323-A - Database request playback method and device based on time axis

CN122019323ACN 122019323 ACN122019323 ACN 122019323ACN-122019323-A

Abstract

The invention relates to the technical field of database performance test and problem diagnosis, and provides a database request playback method and device based on a time axis, wherein the method comprises the steps of loading playback configuration and establishing a time axis mapping relation between recording time and playback time; the method comprises the steps of analyzing historical request data, carrying out request screening and adaptation according to playback configuration and a time axis mapping relation, calculating target trigger time of each request, carrying out scheduling control on the requests, carrying out sequential submitting and executing on the requests according to the target trigger time, collecting request execution results and time-consuming information, carrying out slow SQL judgment and selective result comparison, and outputting a statistical report when termination conditions are met. The invention realizes high-fidelity, controllable, stable and easy-to-verify playback of database request behaviors.

Inventors

  • ZHANG ZETAO
  • Su Zhangyan

Assignees

  • 广州海量数据库技术有限公司

Dates

Publication Date
20260512
Application Date
20260130

Claims (10)

  1. 1. A method for requesting playback of a database based on a timeline, the method comprising: Step S1, loading playback configuration and establishing a time axis mapping relation between recording time and playback time; S2, analyzing historical request data, carrying out request screening and adaptation according to playback configuration and a time axis mapping relation, and calculating target trigger time of each request; Step S3, scheduling control is carried out on the request, and the request is submitted and executed in sequence according to a time sequence relation according to a target trigger moment; And S4, collecting the execution result of the request and the time-consuming information, comparing the slow SQL judgment with the selective result, and outputting a statistical report when the termination condition is met.
  2. 2. The method for requesting playback of a database on a time axis as set forth in claim 1, wherein step S1 comprises loading a playback configuration including a time unit, a speed ratio, a preload window, an upper backlog limit, and a filter rule, normalizing time stamps and duration in a recording stream according to the playback configuration, and establishing a time axis mapping relationship between a recording start time and a playback start time.
  3. 3. The method for requesting playback of a database based on a time axis according to claim 2, wherein in step S1, establishing a time axis mapping relationship between a recording time and a playback time includes: reading time unit parameters of a normalized reference for specifying a time stamp and a duration from a playback configuration; Converting the original time stamp and the original duration of each history request in the record stream into uniform numerical representation according to time units to form normalized time data; Determining a record start time, which is a normalized timestamp of a first processed request in the record stream; and setting the current time of the system or the fixed time designated by the configuration as the playback starting time, and establishing a linear mapping relation between the recording starting time and the playback starting time.
  4. 4. The timeline-based database request playback method according to claim 1, characterized in that step S2 includes: according to the configured white list rule and black list rule, carrying out matching judgment on session identification, user identification or database mode identification contained in the request model, and screening requests which accord with white list conditions and are not excluded by the black list; According to the configured mode mapping table, replacing the mode name of the source end database of the screened request with the mode name corresponding to the target end; For each screened request, calculating the time offset between the normalized timestamp and the recording starting time, dividing the time offset by the configured speed proportion parameter for scaling, and adding the scaled offset to the playback starting time to obtain the target trigger moment of the request on the playback time axis.
  5. 5. The timeline-based database request playback method according to claim 1, characterized in that step S3 includes: The method comprises the steps that a scheduler is used for prefetching and scheduling a request with a target trigger moment in a preloading window range; Continuously monitoring the number of tasks in a task queue to be executed through a scheduler, and when the number exceeds a configured backlog upper limit threshold value, suspending acquiring a new request from upstream until the number of the tasks in the queue falls back below the threshold value, and then resuming prefetching; the requests are grouped according to the session identification or the transaction identification in the request model through a scheduler, and all requests in the same session or transaction group are serially submitted to an executor according to the sequence of the target trigger time.
  6. 6. The timeline-based database request playback method according to claim 1, characterized in that in step S3, the request is executed by the pool of executor threads in the following way: The method comprises the steps that an executive thread pool creates a fixed number of working threads according to a configured parallel pool size parameter during initialization; when the target trigger time of the request arrives, the scheduler packages the request into a task object and submits the task object to a task queue of an executor thread pool; the working thread in the thread pool of the executor acquires a task from the task queue, establishes connection with a target database, executes SQL sentences or database operation instructions corresponding to the request, and acquires an execution result.
  7. 7. The timeline-based database request playback method according to claim 1, characterized in that in step S4, a slow SQL decision is made as follows: recording the actual execution time taken by each request from submitting an executor to returning a result; Comparing the actual execution time consumption with a configured slow SQL decision threshold, and marking the request as slow SQL if the actual execution time consumption exceeds the threshold; Information acquisition is carried out on all marked slow SQL requests, wherein the information acquisition comprises request text, execution time, actual time consumption and session identification, and the request text, the execution time, the actual time consumption and the session identification are stored in a slow SQL record list; After playback is finished, a slow SQL analysis report is generated according to the list, wherein the slow SQL analysis report comprises the quantity of slow SQL, the slowest TOP N request details and time-consuming distribution statistics.
  8. 8. The timeline-based database request playback method according to claim 1, wherein in step S4, the selective result comparison is performed as follows: After the execution result of the request is obtained by the thread pool of the executor, extracting an expected result or a reference result corresponding to the request from the record stream; Comparing the current execution result with the expected result field by field or integrally, wherein the comparison content comprises data content, line number, column structure and execution state; if the difference is detected, recording the difference type, the difference detail, the request identification and the occurrence time, and storing the difference type, the difference detail, the request identification and the occurrence time into a difference record set; after playback is finished, a difference report is generated according to the difference record set, wherein the difference report comprises all requests with differences and specific difference information thereof.
  9. 9. The timeline-based database request playback method according to claim 1, wherein in step S4, the termination condition includes one or a combination of the following: the continuous operation time reaches the upper limit of the configured maximum working time; recording that all requests in the stream are processed, and that no task to be executed or being executed is in the thread pool of the executor; And receiving an active stop instruction sent by a user or an interrupt signal sent by an external system.
  10. 10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1-9 when the program is executed.

Description

Database request playback method and device based on time axis Technical Field The invention relates to the technical field of database performance test and problem diagnosis, in particular to a database request playback method and device based on a time axis. Background In the operation and maintenance, system migration, version upgrading and performance optimization processes of modern databases, the execution process of historical database requests can be truly reproduced. By playing back the real business load, the performance of the database can be effectively evaluated, the compatibility of the new version or the new environment can be verified, the problems on the line can be accurately reproduced, and a reliable basis is provided for problem positioning and solving. Thus, database request playback techniques have become an important tool in database operation, testing, and development. At present, most common database request playback methods are based on a 'queue consumption + fixed rate' implementation mode. The method typically loads a history of requests into a memory queue, fetches requests from the queue and executes at fixed time intervals or at a fixed rate. Although the method is simple to realize and can simulate a certain load pressure, the method has obvious defects in the aspects of truly reproducing the time characteristics and concurrent structures of the source end request, and is particularly characterized in the following aspects: First, existing methods have difficulty maintaining the true temporal distribution and natural concurrency of source requests. In an actual production environment, the arrival of database requests has randomness, burstiness and time sequence correlation, a playback mode with fixed speed cannot restore the real time interval between requests, and the time sequence relation of concurrent requests of a source cannot be accurately simulated. Particularly, under the traffic scene of alternating high and low peaks, fixed-rate playback often causes time sequence distortion, and the actual load characteristics of the system cannot be truly reflected. Second, existing methods lack efficient sequence preservation and backpressure control mechanisms in parallel playback scenarios. If not controlled, parallel playback can easily lead to a confusing order of request execution, especially for requests within the same session or transaction, which can cause data consistency problems or state anomalies. In addition, due to lack of backpressure control, when a large number of requests arrive in a concentrated manner in a short time, the executor is easily submerged in a transient high concurrency manner, so that system resource exhaustion, response delay surge and even service breakdown are caused, and a significant deviation is generated from the actual behavior of the source system. Third, existing schemes have limited capabilities in collaborative processing of recording, parsing, and playback, especially lack of unified management capabilities for global timelines. In a scene requiring recording and playback at the same time or reproducing according to a source end time interval, the existing method often needs to wait for the recording to be completed and then play back, so that real streaming processing cannot be realized, and the end-to-end delay is larger, and the real-time or quasi-real-time verification requirement cannot be met. Fourth, the prior art has significantly shorter boards in terms of functional integrity and configurability. Most playback tools lack a uniform normalized processing of time units (e.g., nanoseconds, microseconds, milliseconds, seconds), and are difficult to accommodate with time recording sources of varying accuracy. Meanwhile, in the aspects of speed proportion adjustment, preloading window control, task backlog management, white list/black list filtering, user/mode filtering, mode mapping, slow SQL judgment, query result comparison and the like, systematic configuration and scheduling support are often lacked, so that the controllability of a playback process is poor, the verification capability is weak, and the adaptability is insufficient. Chinese patent publication No. CN121070977a discloses a method, medium, product and apparatus for playback of database load files, which solves the problem of local compensation of execution time drift, but has significant drawbacks in terms of global timeline management, concurrency control, functional integrity, streaming processing, and systematic design. Therefore, how to provide a database request playback method capable of supporting multiple functions such as time sequence maintenance, concurrency control, backpressure management, dynamic filtering, result verification and the like by taking a unified time axis as a core, so as to realize high-fidelity, controllable and stable reproduction of a source database request behavior, and meet diversified requirements in operation and maintenance