CN-115793971-B - Data processing method, device, server, storage medium and program product

CN115793971BCN 115793971 BCN115793971 BCN 115793971BCN-115793971-B

Abstract

The present application relates to a data processing method, apparatus, server, storage medium, and program product. When a fault of a master log system in a distributed log system is monitored, a target acquires shadow information of data to be written from the log system, the shadow information is sent to a back-end flushing component of the distributed log system, the back-end flushing component is instructed to acquire the data to be written from a temporary disk of the log system according to the shadow information, and the data to be written is written into the back-end disk of the distributed log system. By adopting the method, when any log system in the distributed log system fails, the back-end brushing component reads the data to be written in the fault log system from the temporary disk of the fault-free log system through the shadow information of the data to be written in, and the process can shorten the replay time of the fault-free log system when replaying the data in the temporary disk of the fault log system to the back-end disk.

Inventors

LIU LILI
HAN YONG
KANG HANYU
WU RUIQIANG
ZHEN YU

Assignees

曙光信息产业(北京)有限公司

Dates

Publication Date: 20260508
Application Date: 20221123

Claims (10)

1. A method of data processing, the method comprising: If the fault of a master log system in the distributed log system is monitored, a target slave log system acquires shadow information of data to be written in, wherein the target slave log system is a normal slave log system selected from a plurality of slave log systems after the fault of the master log system; The target sends the shadow information to a back-end brushing component of the distributed log system from the log system, instructs the back-end brushing component to acquire the data to be written from a temporary disk of the log system from the target according to the shadow information, and writes the data to be written into the back-end disk of the distributed log system; The target slave log system responds to a data writing request sent by the master log system and writes data to be written carried in the data writing request into the temporary disk; the target extracts magnetic disk head data from the data to be written from a log system and acquires attribute information of the magnetic disk head data; The target slave log system determines shadow information of the data to be written according to the attribute information of the magnetic disc head data; and the target stores the shadow information of the data to be written into the internal memory of the target from the log system.
2. The method according to claim 1, wherein the obtaining attribute information of the disk head data includes: Acquiring a storage position of the disk head data in the temporary disk and the size and type of the disk head data; And determining attribute information of the magnetic disc head data according to the storage position of the magnetic disc head data and the size and the type of the magnetic disc head data.
3. The method of claim 1, wherein prior to storing the shadow information of the data to be written in its own memory, the method further comprises: detecting whether the residual memory capacity of the memory is larger than or equal to the required memory capacity for storing the shadow information; and if the remaining storage capacity of the memory is greater than or equal to the required storage capacity of the shadow information, executing the step of storing the shadow information of the data to be written into the memory of the memory.
4. A method according to claim 3, characterized in that the method further comprises: If the residual storage capacity of the memory is smaller than the required storage capacity of the shadow information, historical shadow information meeting preset conditions is screened from all the stored shadow information in the memory; And deleting the historical shadow information in the memory, and then executing the step of storing the shadow information of the data to be written into the memory.
5. The method of claim 4, wherein the history shadow information for the preset condition includes any one of: all historical shadow information before the current moment; and presetting all the history shadow information before the history time, wherein the storage amount of all the history shadow information before the history time is larger than or equal to a preset storage amount threshold value.
6. The method according to any one of claims 1-5, wherein the back-end swiping component, after obtaining the data to be written, pre-processes the data to be written to obtain pre-processed data, and then writes the pre-processed data to the back-end disk, wherein the pre-processing includes format conversion and encapsulation.
7. The method of claim 1, wherein after the master log system fails, selecting any one of the plurality of slave log systems that is normal, comprises: The system comprises a plurality of fault-free slave log systems, a slave log system and a distributed coordination service, wherein the fault-free slave log systems are used for electing one slave log system from the fault-free slave log systems based on a preset election strategy, and the preset election strategy comprises at least one of a distributed system election algorithm and the distributed coordination service.
8. A data processing apparatus, the apparatus comprising: the shadow acquisition module is used for acquiring shadow information of data to be written when a fault of a main log system in the distributed log system is monitored; The sending module is used for sending the shadow information to a back-end brushing component of the distributed log system, indicating the back-end brushing component to acquire the data to be written from a temporary disk of the log system according to the shadow information, and writing the data to be written into the back-end disk of the distributed log system; The request response module is used for responding to a data writing request sent by the master journal system, writing data to be written carried in the data writing request into the temporary disk, extracting disk head data from the data to be written, acquiring attribute information of the disk head data, determining shadow information of the data to be written according to the attribute information of the disk head data, and storing the shadow information of the data to be written into a memory of the memory.
9. A server comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1-7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1-7.

Description

Data processing method, device, server, storage medium and program product Technical Field The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, server, storage medium, and program product. Background With the continuous development of big data technology, distributed systems gradually become the mainstream systems of data processing. Among other things, distributed systems often have some special data, e.g., metadata. Such special data needs to meet order retention and transactional properties, and distributed systems are typically implemented using multiple log systems when processing such special data. In order to prevent special data in the log system from being lost under the abnormal condition of the distributed system, in the related art, the log system can read the special data from the high-speed disk and then replay the special data to a back-end disk of the distributed system for permanent storage. But the log system has a long playback time when data is played back to the back-end disk using the related art. Disclosure of Invention In view of the foregoing, it is desirable to provide a data processing method, apparatus, server, storage medium, and program product that can shorten the playback time of a journal system when playing back data to a back-end disk. In a first aspect, the present application provides a data processing method, the method comprising: if the master log system in the distributed log system is monitored to be faulty, the target slave log system acquires the shadow information of the data to be written in; And the target sends the shadow information to a back-end brushing component of the distributed log system from the log system, instructs the back-end brushing component to acquire data to be written from a temporary disk of the log system according to the shadow information, and writes the data to be written into the back-end disk of the distributed log system. In the technical scheme of the embodiment of the application, when the fault of a master log system in a distributed log system is monitored, a target acquires shadow information of data to be written from the log system, the shadow information is sent to a back-end brushing component of the distributed log system, the back-end brushing component is instructed to acquire the data to be written from a temporary disk of the log system according to the shadow information, and the data to be written is written into the back-end disk of the distributed log system. According to the method, when any log system in the distributed log system fails, the back-end brushing component reads the data to be written in the fault log system from the temporary disk of the fault log system through the shadow information of the data to be written in, the process of reading the data to be written in the fault log system from the temporary disk to be written in the back-end disk by the fault log system is avoided, so that the replay time of the fault log system can be shortened when the data in the temporary disk of the fault log system is replayed to the back-end disk, the replay efficiency of the fault log system to the back-end disk is improved, in addition, on the basis of improving the replay efficiency of the fault log system, the temporary disk of the fault log system can be accelerated to provide storage service, the purpose of shortening the cut-off time of the temporary disk in the fault log system is achieved, the input use frequency of the temporary disk in the fault log system is improved, and meanwhile, the fact that the data to be written in the temporary disk is read by the fault log system is prevented from influencing normal operation of other subsequent businesses can be achieved. In one embodiment, the method further comprises: and the target slave log system responds to the data writing request sent by the master log system, writes the data to be written carried in the data writing request into the temporary disk and constructs shadow information of the data to be written. According to the technical scheme, the target slave log system can respond to the data writing request sent by the master log system, write the data to be written carried in the data writing request into the temporary disk and construct the shadow information of the data to be written, so that the back-end brushing component can further read the data to be written from the temporary disk of the fault-free log system through the shadow information of the data to be written, the process that the fault-free log system reads the data to be written of the fault log system in the temporary disk and writes the data to be written into the back-end disk is avoided, and the time for the fault-free log system to read the data to be written is saved. In one embodiment, the target builds shadow information of the data to be written from the log system, including: The target extracts mag