Search

CN-122019600-A - Data processing method and device, storage medium and computer program product

CN122019600ACN 122019600 ACN122019600 ACN 122019600ACN-122019600-A

Abstract

The application discloses a data processing method and device, a storage medium and a computer program product, and relates to the field of data processing, wherein the data processing method comprises the steps of obtaining cache data from a data cache area, wherein the cache data is original data from a preset data source; performing data processing on the cache data to obtain playback data, wherein the playback data has a playback time tag; and distributing the playback data to a data receiving module of a target test environment according to the playback time tag. By adopting the technical scheme, the problem that test data conforming to the production environment cannot be provided quickly is solved.

Inventors

  • WANG SHOUREN
  • XUE QIAN
  • LV QI
  • YE JIARUI

Assignees

  • 中债金融估值中心有限公司

Dates

Publication Date
20260512
Application Date
20251217

Claims (10)

  1. 1. A method of data processing, comprising: Obtaining cache data from a data cache region, wherein the cache data is original data from a preset data source; performing data processing on the cache data to obtain playback data, wherein the playback data has a playback time tag; and distributing the playback data to a data receiving module of a target test environment according to the playback time tag.
  2. 2. The data processing method of claim 1, wherein prior to obtaining the buffered data from within the data buffer, the method further comprises: determining a preset data extraction rule, wherein the data extraction rule comprises connection information and a data extraction range of the preset data source; Performing data extraction operation on the preset data sources according to the preset data extraction rule to obtain heterogeneous data, wherein different data sources in the preset data sources have different data formats, and the heterogeneous data comprise data in different data formats from the preset data sources; and carrying out data format unification operation on the heterogeneous data to obtain the original data set.
  3. 3. The data processing method according to claim 1, wherein performing data processing on the buffered data to obtain playback data comprises: determining a data playback condition of a first data playback task, wherein the data playback condition comprises a data screening rule, a data ordering rule and a data playback time period; screening the cache data according to the data screening rule to obtain target data; sorting the target data according to the data sorting rule to obtain sorting data; And adding a playback time tag for the ordered data according to the data playback time period to obtain the playback data.
  4. 4. The data processing method according to claim 1, wherein the data processing is performed on the buffered data to obtain playback data, further comprising: acquiring a generation time period corresponding to the cache data and a generation time stamp corresponding to each piece of data in the original data; Determining a time conversion rule according to the generation time period and the data playback time period corresponding to the second data playback task; Determining a playback time tag corresponding to the generated time stamp in the data playback time period according to the time conversion rule; and adding the playback time tag to the cache data to obtain the playback data.
  5. 5. The data processing method of claim 1, wherein the data receiving module that distributes the playback data to the target test environment according to the playback time stamp comprises: determining the target test environment with subscription relation with the playback data set according to subscription configuration information; And sending the playback data to a data receiving module of the target test environment in a data stream mode.
  6. 6. The data processing method of claim 5, wherein transmitting the playback data to the data receiving module of the target test environment in a data stream comprises: Determining a release rate mode of the data stream, wherein the release rate mode at least comprises one of an original double-speed mode, an accelerated playback mode and a decelerated playback mode; And sending the playback data to a data receiving module of the target test environment according to the release rate mode of the data stream.
  7. 7. The data processing method of claim 5, wherein transmitting the playback data to the data receiving module of the target test environment in a data stream comprises: generating a plurality of data copies for the playback data in the case where the target test environment includes a plurality of test environments; Distributing a data copy for each test environment, and sending each data copy to a data receiving module of the corresponding test environment in a data stream mode; displaying the sending states and the data processing rates of the multiple data copies to a target object at an interactive interface; and controlling the playback process of the plurality of data copies according to the control instruction of the target object on the interactive interface, wherein the control instruction comprises a stop instruction, a start instruction and a playback speed adjusting instruction.
  8. 8. A data processing apparatus, comprising: the acquisition module is used for acquiring cache data from the data cache area, wherein the cache data is original data from a preset data source; The processing module is used for carrying out data processing on the cache data to obtain playback data, wherein the playback data is provided with a playback time tag; And the distribution module is used for distributing the playback data to the data receiving module of the target test environment according to the playback time tag.
  9. 9. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein the program when run performs the method of any one of claims 1 to 7.
  10. 10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Description

Data processing method and device, storage medium and computer program product Technical Field The present application relates to the field of data processing, and in particular, to a data processing method and apparatus, a storage medium, and a computer program product. Background In the fields of financial transactions and the like involving high concurrency real-time data processing, development and test links of data application are long, requirements of each link are more, when a certain link is changed, test data are frequently and repeatedly sent upstream for logic and technical verification, and real-time data support related work is required in multiple sets of environments such as development, test, user verification, quasi-production and the like. While test data highly conforming to real-time data of the production environment is impossible 724 Are provided at any time. In the prior art, a method for exporting full or incremental data snapshots from a production database through a timing task and then importing the full or incremental data snapshots into a test environment in batches cannot provide data streams in a continuous time window, so that service logic depending on a real-time sequence or state cannot be verified, the data playback precision is in the order of hours or days, and the verification requirement of a millisecond-level service scene cannot be met completely. When test requirements change or specific scenario data is required, database administrators or developers are severely relied upon to manually perform data export, desensitization, conversion, and import operations. This process is highly dependent on manual intervention, and environmental data preparation and switching typically requires configuration time on the order of hours or even days, which is inefficient. Aiming at the problem that test data conforming to the production environment cannot be provided quickly in the related art, no effective solution is proposed at present. Accordingly, there is a need for improvements in the related art to overcome the drawbacks of the related art. Disclosure of Invention The embodiment of the application provides a data processing method and device, a storage medium and a computer program product, which at least solve the problem that test data meeting production environment cannot be provided quickly in the related art. According to one aspect of the embodiment of the application, a data processing method is provided, which comprises the steps of obtaining cache data from a data cache area, wherein the cache data is original data from a preset data source, performing data processing on the cache data to obtain playback data, wherein the playback data is provided with a playback time tag, and distributing the playback data to a data receiving module of a target test environment according to the playback time tag. In an exemplary embodiment, before the cache data is obtained from the data cache area, the method further comprises determining a preset data extraction rule, wherein the data extraction rule comprises connection information and a data extraction range of the preset data source, executing data extraction operation on the preset data source according to the preset data extraction rule to obtain heterogeneous data, wherein different data sources in the preset data source have different data formats, the heterogeneous data comprise data with different data formats from the preset data source, and performing data format unification operation on the heterogeneous data to obtain the original data set. In an exemplary embodiment, the data processing is performed on the cache data to obtain playback data, and the method comprises the steps of determining data playback conditions of a first data playback task, wherein the data playback conditions comprise a data screening rule, a data sorting rule and a data playback time period, screening the cache data according to the data screening rule to obtain target data, sorting the target data according to the data sorting rule to obtain sorting data, and adding a playback time tag to the sorting data according to the data playback time period to obtain the playback data. In an exemplary embodiment, the data processing is performed on the cache data to obtain playback data, and the method further comprises the steps of obtaining a generation time period corresponding to the cache data and a generation time stamp corresponding to each piece of data in the original data, determining a time conversion rule according to the generation time period and a data playback time period corresponding to a second data playback task, determining a playback time label corresponding to the generation time stamp in the data playback time period according to the time conversion rule, and adding the playback time label to the cache data to obtain the playback data. In an exemplary embodiment, the data receiving module for distributing the playback data