CN-117112570-B - Data processing method, apparatus and computer readable storage medium

CN117112570BCN 117112570 BCN117112570 BCN 117112570BCN-117112570-B

Abstract

The embodiment of the application discloses a data processing method, a device and a computer readable storage medium, wherein the method comprises the steps of obtaining a starting instruction aiming at a target database, starting the target database, reading a target table data file and a memory change log file corresponding to the target database stored in a magnetic disk, wherein the memory change log file is used for recording operation information aiming at the total cache data of the target table data file in a memory, the operation information comprises the operated change data in the total cache data, carrying out data merging processing on table data in the target table data file and the change data in the memory change log file to obtain target table data, and constructing target total cache data corresponding to the target table data file in the memory based on the target table data. Therefore, when the database is abnormal and restarted, the latest data of the whole amount of cache in the memory during the database abnormality is recovered, the durability of the data of the whole amount of cache in the memory is ensured, and the data processing efficiency is further improved.

Inventors

Bi Lujun
HUANG XUBIN
CHEN FAN
WANG SHENGYANG
YE ZHANGMING

Assignees

浙江正泰物联技术有限公司

Dates

Publication Date: 20260512
Application Date: 20230913

Claims (8)

1. A method of data processing, comprising: acquiring a starting instruction aiming at a target database, and starting the target database; Reading a target table data file and a memory change log file which are stored in a disk and correspond to the target database, wherein the memory change log file is used for recording operation information of the total cache data of the target table data file in a memory, the operation information comprises change data operated in the total cache data, the target table data file is a table data file adopting the total cache, the memory change log file is a pre-written log file, and a global sequence number exists in each row of data in the target table data file and the memory change log file; acquiring a global sequence number of change data of each line of change data in the memory change log file; Comparing the table data with the change data according to the global sequence number of the change data, and determining target change data which is not written into the target table data file in the change data according to a comparison result; Carrying out data merging processing on the target change data and the table data to obtain target table data; constructing target total cache data corresponding to the target table data file in the memory based on the target table data; determining the file number of the memory change log files; when the number of the files is larger than a preset file number threshold, determining a target memory change log file to be processed according to the file number of the memory change log file, wherein the file number of the target memory change log file is not the largest file number in the file numbers of the memory change log files; judging whether the change data in the target memory change log file is synchronous to a target table data file of the disk or not according to the global sequence number in the target memory change log file; If the table data in the target memory change log file is not synchronized to the target table data file of the disk, synchronizing the table data in the target memory change log file to the target table data file of the disk; and if the target memory change log file is synchronized to the target table data file of the disk, deleting the target memory change log file.
2. The data processing method of claim 1, further comprising: recording the largest target global sequence number in the global sequence numbers of the changed data; After constructing the target total cache data corresponding to the target table data file in the memory based on the target table data, the method further comprises: When a data adding operation aiming at the target total cache data is received, acquiring the maximum target global sequence number of the record; performing sequence number increment processing on the target global sequence number to obtain a first target global sequence number; Writing first target data corresponding to the data adding operation into the target full cache data; and updating the memory change log file based on the first target global sequence number and the first target data.
3. The data processing method of claim 2, wherein the updating the memory change log file based on the first target global sequence number and the first target data comprises: Acquiring the stored data volume of the memory change log file; when the stored data quantity is not greater than a preset data quantity threshold value, writing the first target global sequence number and the first target data into the memory change log file; when the stored data volume is larger than the preset data volume threshold value, acquiring the maximum file number of the memory change log file; creating a new memory change log file, and assigning a file number of the new memory change log file as a file number after the maximum file number is increased; and writing the first target global sequence number and the first target data into the new memory change log file.
4. The data processing method as claimed in claim 1, wherein after said constructing the target full cache data corresponding to the target table data file in the memory based on the target table data, further comprises: Responding to data modification operation aiming at the target full cache data, and positioning second target data corresponding to the data modification operation in the target full cache data; locking the row where the second target data is located, and modifying the second target data based on the data modification operation; And updating the memory change log file based on the global sequence number of the second target data and the modified second target data.
5. The data processing method as claimed in claim 1, wherein after said constructing the target full cache data corresponding to the target table data file in the memory based on the target table data, further comprises: responding to a data deleting operation aiming at the target total cache data, and positioning third target data corresponding to the data deleting operation in the target total cache data; Locking the row where the third target data is located, and modifying the deletion mark corresponding to the third target data; and updating the memory change log file based on the modified deletion mark and the global sequence number corresponding to the third target data.
6. A data processing apparatus, comprising: The database starting unit is used for acquiring a starting instruction aiming at a target database and starting the target database; The file reading unit is used for reading a target table data file and a memory change log file which are stored in a disk and correspond to the target database, wherein the memory change log file is used for recording operation information of the total cache data of a memory for the target table data file, the operation information comprises change data operated in the total cache data, the target table data file is a table data file adopting the total cache, the memory change log file is a pre-written log file, and a global sequence number exists in each row of data in the target table data file and the memory change log file; the data merging unit is used for acquiring the global sequence number of the change data of each line of change data in the memory change log file, comparing the table data with the change data according to the global sequence number of the change data, determining target change data which is not written into the target table data file in the change data according to a comparison result, and carrying out data merging processing on the target change data and the table data to obtain target table data; the data caching unit is used for constructing target total cache data corresponding to the target table data file in the memory based on the target table data; And the data synchronization unit is used for determining the number of files of the memory change log file, determining a target memory change log file to be processed according to the file number of the memory change log file when the number of files is larger than a preset file number threshold, judging whether the change data in the target memory change log file are synchronized into the target table data file of the disk according to the global serial number in the target memory change log file, synchronizing the table data in the target memory change log file into the target table data file of the disk if the number of files is not synchronized into the target table data file of the disk, and deleting the target memory change log file if the number of files is not synchronized into the target table data file of the disk.
7. A computer readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the data processing method of any of claims 1 to 5.
8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the data processing method of any of claims 1 to 5 when the computer program is executed.

Description

Data processing method, apparatus and computer readable storage medium Technical Field The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, and computer readable storage medium. Background In software systems using the relational database management system (MYSQL), there are many tables with small amounts of data that can be cached in full. For example, a Memory storage engine (The Memory Storage Engine, memory storage engine) may be used to cache the table data in full. In the research and practice process of the prior art, the existing data processing method for carrying out full-volume caching on the table data cannot support the persistence of the cached data, and the situation that the data is lost easily occurs when the database is restarted due to abnormality is found, so that the data processing efficiency is lower. Disclosure of Invention The embodiment of the application provides a data processing method, a data processing device and a computer readable storage medium, which can realize that when a database is restarted due to abnormality, the latest data of the whole amount of cache in a memory when the database is abnormal is recovered, thereby ensuring the durability of the data of the whole amount of cache in the memory and further improving the data processing efficiency. The embodiment of the application provides a data processing method, which comprises the following steps: acquiring a starting instruction aiming at a target database, and starting the target database; Reading a target table data file and a memory change log file which are stored in a disk and correspond to the target database, wherein the memory change log file is used for recording operation information of the total cache data of the target table data file in a memory, and the operation information comprises change data operated in the total cache data; Carrying out data merging processing on table data in the target table data file and change data in the memory change log file to obtain target table data; And constructing target total cache data corresponding to the target table data file in the memory based on the target table data. Accordingly, an embodiment of the present application provides a data processing apparatus, including: The database starting unit is used for acquiring a starting instruction aiming at a target database and starting the target database; The file reading unit is used for reading a target table data file and a memory change log file which are stored in a disk and correspond to the target database, wherein the memory change log file is used for recording operation information of the total cache data of the memory aiming at the target table data file, and the operation information comprises the change data operated in the total cache data; The data merging unit is used for carrying out data merging processing on the table data in the target table data file and the change data in the memory change log file to obtain target table data; and the data caching unit is used for constructing target total cache data corresponding to the target table data file in the memory based on the target table data. In an embodiment, a global sequence number exists in each row of data in the target table data file and the memory change log file, and the data merging unit is configured to: acquiring a global sequence number of change data of each line of change data in the memory change log file; Comparing the table data with the change data according to the global sequence number of the change data, and determining target change data which is not written into the target table data file in the change data according to a comparison result; And carrying out data merging processing on the target change data and the table data to obtain target table data. In an embodiment, the data processing apparatus further includes: The target global sequence number recording unit is used for recording the maximum target global sequence number in the global sequence numbers of the changed data; The data processing apparatus further includes: The target global sequence number acquisition unit is used for acquiring the maximum target global sequence number of the record when receiving a data adding operation aiming at the target total cache data; the sequence number increment unit is used for carrying out sequence number increment processing on the target global sequence number to obtain a first target global sequence number; The data writing unit is used for writing first target data corresponding to the data adding operation into the target full cache data; And the first data updating unit is used for updating the memory change log file based on the first target global sequence number and the first target data. In an embodiment, the first data updating unit is configured to: Acquiring the stored data volume of the memory change log file; when the stored data