Search

CN-122019261-A - Data processing method and device

CN122019261ACN 122019261 ACN122019261 ACN 122019261ACN-122019261-A

Abstract

The invention discloses a data processing method and device, and relates to the technical field of cloud computing. The method is applied to a data backup system, the data backup system stores a data processing strategy of target data in advance, the data processing strategy is used for indicating target performance of processing the target data, and the target performance is performance promised to a user by the data backup system. On the basis, after the data backup system detects a target processing task indicating to process target data, the processing task is executed through M execution units, in the process of executing the processing task, the performance of the data backup system for processing the target data is obtained, the performance of the data backup system for processing the target data does not reach the target performance promised for a user, the number of execution units for executing the processing task in parallel is increased, and therefore the performance of the data backup system for processing the target data is improved, the data backup system is enabled to provide the target performance promised for the user for the target data, and the data backup system is enabled to provide the promised performance.

Inventors

  • CHEN KEYUN
  • HUANG JIAQI

Assignees

  • 华为云计算技术有限公司

Dates

Publication Date
20260512
Application Date
20241107

Claims (19)

  1. 1. A data processing method, applied to a data backup system, where the data backup system stores a data processing policy of target data, where the data processing policy of target data is used to indicate a target performance when processing the target data, and the data backup system includes a storage device, where the storage device stores the target data, and the storage device is located in at least one data center, and the method includes: responding to a target processing task, and executing the target processing task through M execution units, wherein M is a positive integer greater than or equal to 1, the target processing task is used for indicating to process the target data, and the processing of the target data comprises the steps of backing up the target data or recovering the target data; acquiring the performance of the data backup system for processing the target data in the process of executing the target processing task; when the performance of the data backup system for processing the target data is smaller than the target performance, N execution units are determined to be added for executing the target processing task, wherein N is a positive integer greater than or equal to 1; And executing the target processing task in parallel through the added N execution units and the M execution units.
  2. 2. The method according to claim 1, wherein, in the case where M is a positive integer greater than 1, The target processing task is executed by M execution units, including that the target processing task is executed in parallel by M execution units; The method further includes reducing a number of execution units that execute the target processing task in parallel when the performance of the data backup system to process the target data is greater than the target performance.
  3. 3. The method according to claim 1 or 2, wherein prior to the execution of the target processing task by M execution units, the method further comprises: Acquiring a first execution rate of the execution unit, wherein the first execution rate is used for indicating the data amount processed in unit time when the execution unit processes the data; and determining the number of the execution units for executing the target processing task to be M according to the first execution rate of the execution units and the target performance.
  4. 4. A method according to any of claims 1-3, wherein a pool of execution units is deployed on the data backup system, the pool of execution units including at least one execution unit, the executing the target processing task by M execution units comprising: and executing the target processing task through the M execution units in the execution unit pool.
  5. 5. The method of any of claims 1-4, wherein the determining to add N execution units for executing the target processing task comprises: Acquiring a second execution rate of the execution unit, wherein the second execution rate is used for indicating the data amount processed in unit time when the execution unit processes the target data; Determining the number of execution units for executing the target processing task to be K according to the second execution rate of the execution units and the target performance, wherein K is a positive integer greater than M; and determining to add N execution units for executing the target processing task, wherein N is the difference value between K and M.
  6. 6. The method according to any one of claims 1 to 5, wherein, The target performance includes a target rate at which the target data is processed, the performance of the data backup system to process the target data includes a rate at which the data backup system processes the target data, or The target performance comprises a target duration for processing the target data, and the performance of the data backup system for processing the target data comprises a duration required by the data backup system for processing the target data.
  7. 7. The method of any of claims 1-6, wherein the data backup system stores a system performance of the data backup system, the system performance being indicative of an amount of data processed per unit time when the data backup system processes data, the method further comprising, prior to the responding to the target processing task: acquiring the data processing strategy, wherein the data processing strategy is used for indicating the performance of a target system when the target data is processed; When the system performance of the data backup system is greater than or equal to the target system performance, storing the data processing strategy, wherein the data processing strategy is used for indicating the data backup system to process the target data based on the data processing strategy; And updating the system performance of the data backup system to obtain the updated system performance of the data backup system, wherein the updated system performance of the data backup system is the difference between the system performance of the data backup system before updating and the target system performance.
  8. 8. The method of claim 7, wherein the step of determining the position of the probe is performed, The system performance of the data backup system comprises the data backup performance of the data backup system and/or the data recovery performance of the data backup system, wherein the data backup performance is used for indicating the data quantity backed up in unit time when the data backup system backs up the data, and the data recovery performance is used for indicating the data quantity recovered in unit time when the data backup system recovers the data; The data backup performance comprises the minimum value of the network bandwidth of the data backup system, the write bandwidth of the data backup system and the data processing rate of the data backup system, and the data recovery performance comprises the minimum value of the network bandwidth of the data backup system, the read bandwidth of the data backup system and the execution rate of the data backup system.
  9. 9. A data processing apparatus, characterized by being applied to a data backup system storing a data processing policy of target data, the data processing policy being used to indicate target performance when processing the target data, the data backup system comprising a storage device storing the target data, the storage device being located in at least one data center, the data processing apparatus comprising: The execution module is used for responding to a target processing task and executing the target processing task through M execution units, wherein M is a positive integer greater than or equal to 1, the target processing task is used for indicating to process the target data, and the processing of the target data comprises the steps of backing up the target data or recovering the target data; The acquisition module is used for acquiring the performance of the data backup system for processing the target data in the process of executing the target processing task; The determining module is used for determining to add N execution units for executing the target processing task when the performance of the data backup system for processing the target data is smaller than the target performance, wherein N is a positive integer greater than or equal to 1; the execution module is further configured to execute the target processing task in parallel through the added N execution units and the M execution units.
  10. 10. The apparatus of claim 9, wherein, in the case where M is a positive integer greater than 1, The execution module is specifically used for executing the target processing task in parallel through M execution units; the determining module is further configured to reduce a number of execution units that execute the target processing task in parallel when the performance of the data backup system to process the target data is greater than the target performance.
  11. 11. The apparatus according to claim 9 or 10, wherein the determining module is further configured to, prior to the execution of the target processing task by the M execution units: Responding to the target processing task, and acquiring a first execution rate of the execution unit, wherein the first execution rate is used for indicating the data amount processed in unit time when the execution unit processes the data; and determining the number of the execution units for executing the target processing task to be M according to the first execution rate of the execution units and the target performance.
  12. 12. The apparatus according to any one of claims 9-11, wherein an execution unit pool is deployed on the data backup system, the execution unit pool including a plurality of execution units, the execution module being specifically configured to: And executing the target processing task through the M execution units of the execution unit pool.
  13. 13. The apparatus according to any one of claims 9-12, wherein the determining module is specifically configured to: Acquiring a second execution rate of the execution unit, wherein the second execution rate is used for indicating the data amount processed in unit time when the execution unit processes the target data; Determining the number of execution units for executing the target processing task to be K according to the second execution rate of the execution units and the target performance, wherein K is a positive integer greater than M; and determining to add N execution units for executing the target processing task, wherein N is the difference value between K and M.
  14. 14. The apparatus of claim 9, wherein the device comprises a plurality of sensors, The target performance includes a target rate at which the target data is processed, the performance of the data backup system to process the target data includes a rate at which the data backup system processes the target data, or The target performance comprises a target duration for processing the target data, and the performance of the data backup system for processing the target data comprises a duration required by the data backup system for processing the target data.
  15. 15. The apparatus of any of claims 9-14, wherein the data backup system stores a system performance of the data backup system, the system performance being indicative of an amount of data processed per unit time when the data backup system processes data, the apparatus further comprising a management module, the management module being configured to, prior to responding to a target processing task: acquiring the data processing strategy, wherein the data processing strategy is used for indicating the performance of a target system when the target data is processed; When the system performance of the data backup system is greater than or equal to the target system performance, storing the data processing strategy, wherein the data processing strategy is used for indicating the data backup system to process the target data based on the data processing strategy; And updating the system performance of the data backup system to obtain the updated system performance of the data backup system, wherein the updated system performance of the data backup system is the difference between the system performance of the data backup system before updating and the target system performance.
  16. 16. The apparatus of claim 15, wherein the device comprises a plurality of sensors, The system performance of the data backup system comprises the data backup performance of the data backup system and/or the data recovery performance of the data backup system, wherein the data backup performance is used for indicating the data quantity backed up by the data backup system in unit time, and the data recovery performance is used for indicating the data quantity recovered in unit time when the data backup system recovers the data; The data backup performance comprises the minimum value of the network bandwidth of the data backup system, the write bandwidth of the data backup system and the data processing rate of the data backup system, and the data recovery performance comprises the minimum value of the network bandwidth of the data backup system, the read bandwidth of the data backup system and the execution rate of the data backup system.
  17. 17. A cluster of computing devices, wherein the cluster of computing devices comprises at least one computing device; Each of the at least one computing device includes a processor, a memory, and computer programs/instructions stored on the memory, the processor of each computing device executing the computer programs/instructions stored in the memory of each computing device to cause each computing device to implement the method of any of claims 1-8.
  18. 18. A computer program product, characterized in that, The computer program product comprising computer programs/instructions which, when executed by a computing device, implement the method of any of claims 1-8.
  19. 19. A computer-readable storage medium comprising, The computer readable storage medium having stored thereon a computer program/instruction which, when executed by a computing device, implements the method of any of claims 1-8.

Description

Data processing method and device Technical Field The present application relates to the field of cloud computing technologies, and in particular, to a data processing method and apparatus. Background Currently, in order to prevent data loss, data is often backed up using a data backup system. The related art adopts a backup scheme that a highest rate is preset for data to be backed up, and the rate of the data backup system when executing a backup task for the data to be backed up cannot exceed the highest rate. However, the backup scheme can avoid that the backup task of single data occupies the whole performance of the data backup system, so that the backup tasks of a plurality of data can be smoothly executed at the same time, but as the requirement of users on the reliability of the backup performance is higher and higher, the scheme in the related art cannot meet the requirements of the users. Disclosure of Invention The application provides a data processing method and device, which can provide promised processing performance for users. The first aspect provides a data processing method applied to a data backup system, wherein the data backup system stores a data processing strategy of target data, the data processing strategy is used for indicating target performance when the target data is processed, the data backup system comprises a storage device, the storage device stores the target data, the storage device is located in at least one data center, the method comprises the steps of responding to a target processing task, executing the target processing task through M execution units, wherein M is a positive integer greater than or equal to 1, the target processing task is used for indicating the target data to be processed, the target data to be processed comprises backup target data or recovery target data, in the process of executing the target processing task, acquiring the performance of the data backup system for processing the target data, determining that the performance of the data backup system for processing the target data is smaller than the target performance, adding N execution units for executing the target processing task, wherein N is a positive integer greater than or equal to 1, and executing the target processing task in parallel through the added N execution units and M execution units. In the above-mentioned scheme, the data backup system stores in advance a data processing policy of target data, where the data processing policy is used to indicate a target performance when processing the target data, and the target performance is a processing performance promised by the data backup system to a user, in other words, a performance that can be achieved when the user expects the data backup system to process the target data. On the basis, after the data backup system detects the target processing task indicating to process the target data, the target processing task is executed through M execution units, in the process of executing the target processing task, the performance of the data backup system for processing the target data is obtained, whether the performance of the data backup system for processing the target data reaches the processing performance promised for a user or not is judged, the data backup system determines that N execution units are added for executing the target processing task, and the target processing task is executed in parallel through N+M execution units, so that the performance of the data backup system for processing the target data is improved, the performance of the data backup system for processing the target data reaches the processing performance promised for the user, and further the data backup system is enabled to provide the promised processing performance for the user, for example, the processing performance can be the data backup performance when the target data is backed up, the data recovery performance when the target data is recovered, and the like. In one possible implementation, where M is a positive integer greater than 1, performing the target processing task by the M execution units includes performing the target processing task in parallel by the M execution units, and the method further includes reducing the number of execution units performing the target processing task in parallel when the performance of the data backup system to process the target data is greater than the target performance. In the implementation manner, when the performance of the data backup system for processing the target data is greater than the processing performance promised for the user, the number of execution units for executing the processing tasks in parallel is reduced, so that the performance of the data backup system for processing the target data can be reduced, and further the method is beneficial to reducing the computing resources occupied by the target processing tasks on the basis of guaranteeing the promised processing performance for