CN-122001933-A - Data processing method and related equipment

Abstract

The embodiment of the application provides a data processing method and related equipment, the method is applied to a data processing system, the data processing system comprises a content distribution network and a server, the server comprises a main service cluster and a playback service cluster, the main service cluster and the playback service cluster respectively correspond to different cluster information, and the method comprises the steps of sending a task pulling request to the main service cluster if the target service cluster indicated by target cluster information included in task information of an update task is obtained as the main service cluster, so that the main service cluster obtains update data of the update task based on task identification information in the task pulling request; if the target service cluster is a playback service cluster, a task pulling request is sent to the playback service cluster so that the playback service cluster can acquire update data of an update task based on task identification information in the task pulling request, and the cache data in the node equipment is updated according to the update data returned by the target service cluster, so that the task pulling efficiency can be improved.

Inventors

• Zhou Zezhan

Assignees

• 腾讯科技（深圳）有限公司

Dates

Publication Date

20260508

Application Date

20241108

Claims (20)

1. 1. A data processing method is characterized in that the method is applied to a data processing system, the data processing system comprises a content distribution network and a service end, the content distribution network comprises node equipment, the service end comprises a main service cluster and a playback service cluster, the main service cluster and the playback service cluster respectively correspond to different cluster information, and the method comprises the following steps: Task information of an update task is obtained, wherein the task information comprises target cluster information and task identification information of the update task; If the target service cluster indicated by the target cluster information is the main service cluster, a task pulling request is sent to the main service cluster, so that the main service cluster acquires update data of the update task based on task identification information in the task pulling request; If the target service cluster indicated by the target cluster information is the playback service cluster, a task pulling request is sent to the playback service cluster, so that the playback service cluster acquires update data of the update task based on task identification information in the task pulling request; And receiving the update data of the update task returned by the target service cluster, and updating the cache data in the node equipment according to the update data.
2. 2. The method of claim 1, wherein the primary service cluster comprises a primary access object and a primary data proxy object, the primary access object and the primary data proxy object being respectively deployed at one or more server instances in the primary service cluster; The sending a task pulling request to the main service cluster, so that the main service cluster obtains update data of the update task based on task identification information in the task pulling request, including: A task pulling request is sent to a server instance corresponding to the main access object through the main access object, the task pulling request is used for triggering the server instance corresponding to the main access object to carry out combination processing on the task pulling request, the combined request is sent to the main data proxy object through the main access object, and the server instance corresponding to the main data proxy object acquires update data of the update task based on task identification information included in the combined request; the receiving the update data of the update task returned by the target service cluster comprises the following steps: And receiving the update data of the update task returned by the main access object, wherein the update data returned by the main access object is acquired from the server instance corresponding to the main data proxy object by the main access object.
3. 3. The method of claim 1, wherein the obtaining task information of the update task comprises: a task information acquisition request is sent to the main service cluster, and the task information acquisition request is used for triggering the main service cluster to acquire task information of an update task from a central control module of the server; Receiving task information of the update task returned by the main service cluster; The target cluster information in the task information is determined by a decision module in the server according to the node association information of the node equipment.
4. 4. The method of claim 1, wherein the server side sets a load balancer, which provides a virtual service address of a service for the node device; and if the target service cluster indicated by the target cluster information is the main service cluster, sending a task pulling request to the main service cluster, including: And sending a task pulling request to the load balancer according to the virtual service address, so that the load balancer forwards the task pulling request to the server instance in the main service cluster according to the load information of each server instance after determining that the target service cluster indicated by the target cluster information is the main service cluster.
5. 5. A data processing method is characterized in that the method is applied to a data processing system, the data processing system comprises a content distribution network and a server, the content distribution network comprises node equipment, the server comprises at least two service clusters, each service cluster corresponds to different cluster information, and a target service cluster in the at least two service clusters comprises a target access object and a target data proxy object, and the method comprises the following steps: Receiving a task pulling request sent by the node equipment based on target cluster information through the target access object, wherein the task pulling request comprises task identification information of an update task; Merging the task pulling requests through the server instance corresponding to the target access object, and sending the merged requests to the target data proxy object through the target access object, wherein the merged requests comprise the task identification information; Acquiring update data of an update task based on task identification information in the combined request through a server instance corresponding to the target data proxy object, and returning the update data of the update task to the target access object; and returning the update data of the update task to the node equipment through the target access object.
6. 6. The method of claim 5, wherein the number of the merged requests is a plurality, the obtaining, by the server instance corresponding to the target data proxy object, update data of the update task based on task identification information in the merged requests includes: Combining the multiple combined requests through the server instance corresponding to the target data proxy object to obtain a combined request, wherein the combined request comprises the task identification information; sending the merging request to a storage layer through the target data proxy object, wherein the merging request is used for triggering the storage layer to return update data of an update task based on the task identification information; and acquiring the update data of the update task returned by the storage layer through the target data proxy object.
7. 7. The method of claim 5, wherein the at least two service clusters comprise a primary service cluster, the target service cluster being the primary service cluster, the method further comprising: Receiving a task information acquisition request sent by the node equipment through the target access object; responding to the task information acquisition request, and acquiring task identification information and target cluster information of an update task to be pulled by the node equipment; And determining the task information of the update task by the target cluster information and the task identification information, and returning the task information of the update task to the node equipment.
8. 8. The method of claim 7, wherein the obtaining task identification information and target cluster information of an update task to be pulled by the node device comprises: acquiring task identification information and target cluster information of an update task to be pulled by the node equipment from a central control module of the server through the target access object; The target cluster information in the task information is determined by a decision module in the server according to the node association information of the node equipment.
9. 9. The method of claim 5, wherein the server sets a load balancer, the load balancer provides a virtual service address of a service for the node device, and the receiving, by the target access object, a task pull request sent by the node device based on target cluster information includes: And receiving a task pulling request forwarded by a load balancer based on target cluster information through the target access object, wherein the task pulling request is sent to the load balancer by the node equipment based on the virtual service address.
10. 10. A data processing method is characterized in that the method is applied to a data processing system, the data processing system comprises a content distribution network and a server, the content distribution network comprises node equipment, the server comprises at least two service clusters, each service cluster corresponds to different cluster information, and the method comprises the following steps: acquiring node association information of the node equipment; determining target cluster information of a target service cluster for acquiring an update task from cluster information corresponding to the at least two service clusters according to the node association information; And sending the target cluster information to a central control module of the server side so that the central control module associates the target cluster information with the node equipment.
11. 11. The method of claim 10, wherein the server has recorded a creation time of an object creation update task, the node association information includes an initiation time of the task information acquisition request, the at least two service clusters include a main service cluster and a playback service cluster; The determining, according to the node association information, target cluster information of a target service cluster for acquiring an update task from cluster information corresponding to the at least two service clusters includes: Determining the equipment type of the node equipment according to the creation time and the initiation time of the update task; if the equipment type of the node equipment is the first equipment type, determining a target service cluster for acquiring an update task as the main service cluster, and determining cluster information of the main service cluster as the target cluster information; And if the equipment type of the node equipment is the second equipment type, determining a target service cluster for acquiring the update task as the playback service cluster, and determining the cluster information of the playback service cluster as the target cluster information.
12. 12. The method of claim 10, wherein the server is provided with a white list, the white list comprises a plurality of address information, each address information corresponds to one device, the node association information comprises target address information, and the at least two service clusters comprise a main service cluster and a playback service cluster; The determining, according to the node association information, target cluster information of a target service cluster for acquiring an update task from cluster information corresponding to the at least two service clusters includes: if the target address information is located in the white list, determining a target service cluster for acquiring an update task as the main service cluster, and determining cluster information of the main service cluster as the target cluster information; And if the target address information is not in the white list, determining a target service cluster for acquiring an update task as the playback service cluster, and determining the cluster information of the playback service cluster as the target cluster information.
13. 13. The method of claim 10, wherein the node-associated information comprises the node device reporting heartbeat information over a historical period of time, the at least two service clusters comprising a primary service cluster and a playback service cluster; The determining, according to the node association information, target cluster information of a target service cluster for acquiring an update task from cluster information corresponding to the at least two service clusters includes: According to the heartbeat information reported by the node equipment in the historical time period, determining the equipment type of the node equipment; if the equipment type of the node equipment is the first equipment type, determining a target service cluster for acquiring an update task as the main service cluster, and determining cluster information of the main service cluster as the target cluster information; And if the equipment type of the node equipment is the second equipment type, determining a target service cluster for acquiring the update task as the playback service cluster, and determining the cluster information of the playback service cluster as the target cluster information.
14. 14. The method of claim 10, wherein the number of node devices is one or more, the update task is stored in a storage layer, the method further comprising: When each node device needs to acquire the update data of the update task, searching the distributed lock in the storage layer according to the lock name of the distributed lock; If the distributed lock is found, generating a unique identifier corresponding to each node device according to the address information of each node device and the task identification information of the update task; storing unique identifiers corresponding to the node devices into the storage layer as the values of the distributed locks respectively, and After the update data of the update task acquired by the target service cluster, distributing the update data of the update task acquired by the target service cluster to a plurality of node devices by utilizing an asynchronous coroutine; and deleting the distributed lock when the update data distribution of the update task is completed.
15. 15. The data processing device is characterized by being applied to a data processing system, wherein the data processing system comprises a content distribution network and a service end, the content distribution network comprises node equipment, the service end comprises a main service cluster and a playback service cluster, the main service cluster and the playback service cluster respectively correspond to different cluster information, and the device comprises: The processing unit is used for acquiring task information of an update task, wherein the task information comprises target cluster information and task identification information of the update task; The processing unit is further configured to send a task pulling request to the main service cluster if the target service cluster indicated by the target cluster information is the main service cluster, so that the main service cluster obtains update data of the update task based on task identification information in the task pulling request; the processing unit is further configured to send a task pulling request to the playback service cluster if the target service cluster indicated by the target cluster information is the playback service cluster, so that the playback service cluster obtains update data of the update task based on task identification information in the task pulling request; And the receiving unit is used for receiving the update data of the update task returned by the target service cluster and updating the cache data in the node equipment according to the update data.
16. 16. A data processing device is characterized in that the device is applied to a data processing system, the data processing system comprises a content distribution network and a service end, the content distribution network comprises node equipment, the service end comprises at least two service clusters, each service cluster corresponds to different cluster information, and a target service cluster in the at least two service clusters comprises a target access object and a target data agent object, and the device comprises: The receiving unit is used for receiving a task pulling request sent by the node equipment based on target cluster information through the target access object, wherein the task pulling request comprises task identification information of an update task; The processing unit is used for merging the task pulling requests through the server instance corresponding to the target access object and sending the merged requests to the target data proxy object through the target access object, wherein the merged requests comprise the task identification information; the processing unit is further configured to obtain, through a server instance corresponding to the target data proxy object, update data of an update task based on task identification information in the combined request, and return the update data of the update task to the target access object; and the processing unit is also used for returning the update data of the update task to the node equipment through the target access object.
17. 17. The data processing device is characterized by being applied to a data processing system, wherein the data processing system comprises a content distribution network and a service end, the content distribution network comprises node equipment, the service end comprises at least two service clusters, each service cluster corresponds to different cluster information, and the device comprises: an obtaining unit, configured to obtain node association information of the node device; the processing unit is used for determining target cluster information of a target service cluster for acquiring an update task from cluster information corresponding to the at least two service clusters according to the node association information; The processing unit is further configured to send the target cluster information to a central control module of the server, so that the central control module associates the target cluster information with the node device.
18. 18. A computer device, comprising: A processor adapted to execute a computer program; A computer readable storage medium having stored therein a computer program which, when executed by the processor, performs the data processing method of any of claims 1-14.
19. 19. A computer-readable storage medium, characterized in that the computer storage medium stores a computer program which, when executed by a processor, performs the data processing method of any of claims 1-14.
20. 20. A computer program product, characterized in that the computer program product comprises a computer program which, when executed by a processor, implements the data processing method of any of claims 1-14.

Description

Data processing method and related equipment Technical Field The application relates to the technical field of Internet, in particular to a data processing method and related equipment. Background With the development of internet technology, a corresponding refreshing client and a cache service are installed before each node device in the content distribution network is on line. The refreshing client smoothly submits the refreshing task of the node equipment to the cache service of the node equipment by continuously requesting the refreshing task of the node equipment from the service end so as to delete the old cache of the current network. Currently, each time a refresh client requests a server, at most 1 ten thousand tasks are pulled (if the number of refresh tasks is less than 1 ten thousand, all existing refresh tasks are pulled), and then the tasks are smoothly submitted to a local cache service. When network anomalies such as poor network quality exist, the task pulling speed of the refreshing client can be obviously slowed down, even no task is pulled, so that once the network is recovered, a large number of tasks which can not be pulled in time by the refreshing client are pulled to the server through a single channel, and the problem of low task pulling efficiency exists. Disclosure of Invention The embodiment of the application provides a data processing method and related equipment, which can improve the task pulling efficiency. In one aspect, an embodiment of the present application provides a data processing method, where the method is applied to a data processing system, the data processing system includes a content distribution network and a server, the content distribution network includes a node device, the server includes a main service cluster and a playback service cluster, and the main service cluster and the playback service cluster correspond to different cluster information respectively, and the method includes: task information of an update task is obtained, wherein the task information comprises target cluster information and task identification information of the update task; If the target service cluster indicated by the target cluster information is a main service cluster, a task pulling request is sent to the main service cluster, so that the main service cluster can acquire update data of an update task based on task identification information in the task pulling request; If the target service cluster indicated by the target cluster information is a playback service cluster, a task pulling request is sent to the playback service cluster, so that the playback service cluster acquires update data of an update task based on task identification information in the task pulling request; and receiving the update data of the update task returned by the target service cluster, and updating the cache data in the node equipment according to the update data. In one aspect, an embodiment of the present application provides a data processing method, where the method is applied to a data processing system, the data processing system includes a content distribution network and a server, the content distribution network includes a node device, the server includes at least two service clusters, each service cluster corresponds to different cluster information, and a target service cluster in the at least two service clusters includes a target access object and a target data proxy object, where the method includes: receiving a task pulling request sent by node equipment based on target cluster information through a target access object, wherein the task pulling request comprises task identification information of an update task; Combining the task pulling requests through the server instance corresponding to the target access object, and sending the combined requests to the target data proxy object through the target access object, wherein the combined requests comprise task identification information; Acquiring update data of an update task based on task identification information in the combined request through a server instance corresponding to the target data proxy object, and returning the update data of the update task to the target access object; And returning the update data of the update task to the node equipment through the target access object. In one aspect, an embodiment of the present application provides a data processing method, where the method is applied to a data processing system, the data processing system includes a content distribution network and a server, the content distribution network includes a node device, the server includes at least two service clusters, each service cluster corresponds to different cluster information, and the method includes: acquiring node association information of node equipment; Determining target cluster information of a target service cluster for acquiring an update task from cluster information corresponding to at least two service clus