CN-118827696-B - Data synchronization method, device, equipment, storage medium and product

CN118827696BCN 118827696 BCN118827696 BCN 118827696BCN-118827696-B

Abstract

The application discloses a data synchronization method, a device, equipment, a storage medium and a product. The method comprises the steps of receiving a first file list to be synchronized sent by a synchronization control server, wherein the first file list to be synchronized comprises a plurality of file information, selecting first file information from the plurality of file information, sending a first request carrying the first file information to a cache server, synchronizing the first file to the file server when the synchronization state of the first file fed back based on the first request is that the synchronization is not performed by other synchronization servers except the first synchronization server in the plurality of synchronization servers, and returning to execute the step of selecting the first file information from the plurality of file information when the synchronization of the first file is successful until the files corresponding to the plurality of file information are synchronized to the file server. The embodiment of the application improves the synchronization efficiency of a large number of files.

Inventors

JIANG JIAJIA

Assignees

中移物联网有限公司
中国移动通信集团有限公司

Dates

Publication Date: 20260512
Application Date: 20240530

Claims (15)

1. A data synchronization method, applied to a first synchronization server, the first synchronization server being any synchronization server among a plurality of synchronization servers in a data synchronization system, the data synchronization system further comprising a synchronization control server and a cache server, the method comprising: Receiving a first file list to be synchronized sent by the synchronization control server, wherein the first file list to be synchronized comprises a plurality of file information; selecting first file information from the plurality of file information; Sending a first request carrying first file information to the cache server, so that the cache server determines the synchronization state of a first file corresponding to the first file information based on a second file to be synchronized, wherein the second file to be synchronized is the first file to be synchronized in the cache server, the second file to be synchronized and the first file to be synchronized have the same file information, and the first request is used for acquiring the synchronization state of the first file corresponding to the first file information; synchronizing the first file to a file server when the synchronization state of the first file received based on the first request feedback is not synchronized by other synchronization servers than the first synchronization server among the plurality of synchronization servers; returning to execute the step of selecting the first file information from the plurality of file information until the files corresponding to the plurality of file information are synchronized to the file server under the condition that the first file is successfully synchronized; the method further comprises the steps of: In the process of synchronizing the first file, locking second file information in a second file list to be synchronized in the cache server according to preset time, so that the synchronization state of the file corresponding to the second file information is modified into synchronization, wherein the second file information is identical to the first file information; The second file list to be synchronized is sent to the cache server by the synchronization control server, and the second file list to be synchronized and the first file list to be synchronized have the same file information.
2. The method for synchronizing data according to claim 1, wherein after the sending of the first request carrying the first file information to the cache server, the method further comprises: and returning to execute the step of selecting the first file information from the plurality of file information until the first file corresponding to the first file information is not synchronized by other synchronization servers except the first synchronization server in the plurality of synchronization servers under the condition that the synchronization state of the first file fed back based on the first request is synchronized by the other synchronization servers except the first synchronization server in the plurality of synchronization servers.
3. The data synchronization method as claimed in claim 1, wherein said returning the step of selecting the first file information from the plurality of file information in case that the first file synchronization is successful comprises: Under the condition that the first file is successfully synchronized, acquiring a first check file in the first file which is successfully synchronized in the file server; comparing the first check file with a second check file of the first file recorded in the first synchronization server; And under the condition that the first check file is the same as the second check file, sending a synchronization success instruction to the cache server, and returning to execute the step of selecting the first file information from the plurality of file information.
4. The data synchronization method of claim 1, wherein after synchronizing the first file to a file server, the method further comprises: recording breakpoint information of the first file under the condition that the first file fails to be synchronized; and sending a synchronization failure instruction containing the breakpoint information to the cache server.
5. The data synchronization method according to claim 4, wherein after the sending of the synchronization failure instruction containing the breakpoint information to the cache server, the method further comprises: Receiving a failure processing list sent by the synchronous control server, wherein the failure processing list is generated by the cache server based on the synchronous failure instruction and is sent to the synchronous control server; determining a breakpoint position of the first file according to the breakpoint information in the failure processing list, wherein the breakpoint position is a position of interrupt synchronization in the first file; and at the breakpoint position, synchronizing the first file to the file server.
6. The data synchronization method according to claim 1, wherein the synchronization state of the first file is determined not to be synchronized by other synchronization servers than the first synchronization server among the plurality of synchronization servers in the case that a distributed lock based on the first request feedback is received.
7. A data synchronization method, applied to a cache server, where the cache server exists in a data synchronization system, and the data synchronization system further includes a synchronization control server and a plurality of synchronization servers, the method comprising: Receiving a first request carrying first file information sent by a first synchronization server, wherein the first request is used for acquiring a synchronization state of a first file corresponding to the first file information, the first file information is file information selected by the first synchronization server from a first file list to be synchronized including a plurality of file information, and the first file list to be synchronized is a file list sent by the first synchronization server; Determining a synchronization state of a first file corresponding to the first file information based on a second file to be synchronized, wherein the first synchronization server is any one of the plurality of synchronization servers, the second file to be synchronized is a first file to be synchronized in the cache server, and the second file to be synchronized and the first file to be synchronized have the same file information; the method comprises the steps of sending a first file to a first synchronization server, sending the synchronization state of the first file to the first synchronization server, enabling the first synchronization server to synchronize the first file to the file server when the synchronization state of the first file fed back based on a first request is received and is not synchronized by other synchronization servers except the first synchronization server in the plurality of synchronization servers, and returning to execute the step of selecting the first file information in the plurality of file information when the first file is successfully synchronized until files corresponding to the plurality of file information are synchronized to the file server, and locking second file information in a second file to be synchronized list in the cache server according to preset time in the process of synchronizing the first file, so that the synchronization state of the files corresponding to the second file information is modified to be synchronized, wherein the second file to be synchronized list is the file to be synchronized, and the second file to be synchronized is the same as the first file information, and the first file to be synchronized list is sent to the synchronization controller.
8. The data synchronization method of claim 7, wherein the method further comprises: and deleting second file information which is the same as the first file information in the second file to be synchronized under the condition that a synchronization success instruction which is sent by the first synchronization server and used for indicating that the first file verification is successful is received.
9. The data synchronization method of claim 7, wherein the method further comprises: Receiving a synchronization failure instruction sent by the first synchronization server, wherein the synchronization failure instruction comprises breakpoint information of the first file; recording the breakpoint information into a failure processing list; And sending the failure processing list to the synchronous control server.
10. The data synchronization method according to claim 9, wherein after the sending of the failure handling list to the synchronization control server, the method further comprises: And under the condition that file information does not exist in the second file to be synchronized, sending a second request to the synchronous control server so that the synchronous control server sends the failure processing list to the first synchronous server.
11. A data synchronization device, the device comprising: the first receiving module is used for receiving a first file list to be synchronized, which is sent by the synchronization control server and comprises a plurality of file information; The selecting module is used for selecting first file information from the plurality of file information; The first sending module is used for sending a first request carrying first file information to the cache server so that the cache server can determine the synchronous state of a first file corresponding to the first file information based on a second file to be synchronized, wherein the second file to be synchronized is the first file to be synchronized in the cache server, the second file to be synchronized and the first file to be synchronized have the same file information, and the first request is used for acquiring the synchronous state of the first file corresponding to the first file information; A second sending module, configured to synchronize the first file to a file server when the synchronization status of the first file that is received based on the first request feedback is that the synchronization is not performed by other synchronization servers than the first synchronization server among the plurality of synchronization servers; The execution module is used for returning to execute the step of selecting the first file information from the plurality of file information until the files corresponding to the plurality of file information are synchronized to the file server under the condition that the first file is successfully synchronized; The device is specifically configured to lock second file information in a second file list to be synchronized in the cache server according to a preset time in the process of synchronizing the first file, so that a synchronization state of a file corresponding to the second file information is modified to be synchronous, wherein the second file information is identical to the first file information, the second file list to be synchronized is sent to the cache server by the synchronization control server, and the second file list to be synchronized and the first file list to be synchronized have identical file information.
12. A data synchronization device, the device comprising: the second receiving module is used for receiving a first request carrying first file information sent by a first synchronization server, wherein the first request is used for acquiring the synchronization state of a first file corresponding to the first file information, the first file information is file information selected by the first synchronization server from a plurality of file information in a first file list to be synchronized, and the first file list to be synchronized is a file list sent by the first synchronization server; The first determining module is configured to determine a synchronization state of a first file corresponding to the first file information based on a second file to be synchronized, where the first synchronization server is any one of the plurality of synchronization servers, the second file to be synchronized is a first file to be synchronized in the cache server, and the second file to be synchronized and the first file to be synchronized have the same file information; And a third sending module, configured to send the synchronization status of the first file to the first synchronization server, so that the first synchronization server synchronizes the first file to the file server when receiving the synchronization status of the first file fed back based on the first request and not being synchronized by other synchronization servers except the first synchronization server in the plurality of synchronization servers, and return to execute the step of selecting the first file information from the plurality of file information when the first file is successfully synchronized until the files corresponding to the plurality of file information are synchronized to the file server, and lock the second file information in a second file to be synchronized list in the cache server according to a preset time in the process of synchronizing the first file, so that the synchronization status of the file corresponding to the second file information is modified to be in synchronization, where the second file information is the same as the first file information, and the second file to be synchronized is the file to be synchronized in the first cache list.
13. A terminal device comprising a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a data synchronization method as claimed in any one of claims 1-10.
14. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon computer program instructions, which when executed by a processor, implement a data synchronization method according to any of claims 1-10.
15. A computer program product, characterized in that the computer product comprises a computer program which, when executed by a processor, implements the data synchronization method of any one of claims 1-10.

Description

Data synchronization method, device, equipment, storage medium and product Technical Field The present application relates to the field of data processing technologies, and in particular, to a data synchronization method, apparatus, device, storage medium, and product. Background In the prior art, data synchronization can be performed through two redis (Remote Dictionary Server, remote dictionary service) cache servers, wherein file information in a backup server is recorded in one redis server, file information in a file server is recorded in the other redis server, file differences are checked through a redis comparison program, and a file synchronization program is informed of the file differences, and then the file synchronization program synchronizes the difference files. Through the implementation mode, incremental synchronization can be performed for each synchronization, and redundant synchronization is prevented from wasting bandwidth and resources. However, the prior art is in a single-node mode, and the synchronization efficiency is lower for large-batch file synchronization. Disclosure of Invention The embodiment of the application provides a data synchronization method, a device, equipment, a storage medium and a product, which are used for solving the problem of low synchronization efficiency of large-batch file synchronization. In a first aspect, an embodiment of the present application provides a data synchronization method, applied to a first synchronization server, where the first synchronization server is any synchronization server of a plurality of synchronization servers in a data synchronization system, and the data synchronization system further includes a synchronization control server and a cache server, where the method includes: Receiving a first file list to be synchronized sent by the synchronization control server, wherein the first file list to be synchronized comprises a plurality of file information; selecting first file information from the plurality of file information; sending a first request carrying first file information to the cache server, wherein the first request is used for acquiring the synchronous state of a first file corresponding to the first file information; synchronizing the first file to a file server when the synchronization state of the first file received based on the first request feedback is not synchronized by other synchronization servers than the first synchronization server among the plurality of synchronization servers; and under the condition that the first file is successfully synchronized, returning to the step of selecting the first file information from the plurality of file information until the files corresponding to the plurality of file information are synchronized into the file server. In a second aspect, an embodiment of the present application provides a data synchronization method, applied to a cache server, where the cache server exists in a data synchronization system, and the data synchronization system further includes a synchronization control server and a plurality of synchronization servers, where the method includes: Receiving a first request carrying first file information sent by a first synchronous server; determining a synchronization state of a first file corresponding to the first file information based on a second file to be synchronized, wherein the first synchronization server is any one of the plurality of synchronization servers; and sending the synchronous state of the first file to the first synchronous server. In a third aspect, an embodiment of the present application provides a data processing apparatus, the apparatus including: the first receiving module is used for receiving a first file list to be synchronized, which is sent by the synchronization control server and comprises a plurality of file information; The selecting module is used for selecting first file information from the plurality of file information; The first sending module is used for sending a first request carrying first file information to the cache server, wherein the first request is used for acquiring the synchronous state of a first file corresponding to the first file information; A second sending module, configured to synchronize the first file to a file server when the synchronization status of the first file that is received based on the first request feedback is that the synchronization is not performed by other synchronization servers than the first synchronization server among the plurality of synchronization servers; And the execution module is used for returning to execute the step of selecting the first file information from the plurality of file information until the files corresponding to the plurality of file information are synchronized to the file server under the condition that the first file is successfully synchronized. In a fourth aspect, an embodiment of the present application provides a data processing apparatus,