Search

CN-114037090-B - Distributed vector processing method, device, system and medium

CN114037090BCN 114037090 BCN114037090 BCN 114037090BCN-114037090-B

Abstract

The invention discloses a distributed vector processing method, a device, a system and a medium, wherein the system comprises at least two clusters, each cluster comprises a plurality of nodes, and the number of the nodes in each cluster in at least two clusters is the same. In each cluster, the plurality of nodes comprises two nodes, namely an aggregation node and a calculation node. The aggregation nodes in each cluster are used for managing data in the cluster where the aggregation nodes are located, and the calculation nodes are used for storing and calculating the data of at least one partition. According to the scheme provided by the invention, each computing node in each cluster can store and calculate data in parallel, a local algorithm can be quickly transplanted without a distributed foundation, distributed calculation is realized, and the development efficiency of a federal algorithm can be accelerated.

Inventors

  • LIU YUDE
  • HUANG QIJUN
  • CHEN RUIQIN

Assignees

  • 深圳前海微众银行股份有限公司

Dates

Publication Date
20260512
Application Date
20211109

Claims (20)

  1. 1. A distributed vector processing system, comprising: The system comprises at least two clusters, wherein each cluster comprises a plurality of nodes, and the number of the nodes in each cluster is the same; in each cluster, the plurality of nodes comprise two nodes, namely an aggregation node and a calculation node; The aggregation nodes in each cluster are used for managing data in the cluster where the aggregation nodes are located, and the calculation nodes are used for storing and calculating the data of at least one partition; wherein, the aggregation nodes in each cluster are specifically used for: Acquiring a calculation instruction, wherein the calculation instruction is used for indicating to calculate the data vector in the cluster and comprises an operation type and a data identifier; Determining an operation instruction according to the operation type and the data identifier, wherein the operation instruction comprises an operator and a vector key value; sending the operation instruction to each computing node in the cluster respectively; accordingly, the computing nodes in each cluster are specifically configured to: receiving an operation instruction sent by an aggregation node of the cluster, wherein the operation instruction comprises an operator and a vector key value; according to the operator, carrying out operation on the data vector corresponding to the vector key value in at least one local partition to obtain an operated data vector; Wherein the aggregation nodes in each cluster are further configured to: splitting the received data to be stored according to the number of the data partitions in the cluster to obtain a plurality of data vectors; Determining a vector key value corresponding to each data vector according to a preset rule; According to the number of the data partitions on each computing node, the plurality of data vectors and the vector key values are respectively sent to each computing node in the cluster; accordingly, the computing nodes in each cluster are also to: receiving at least one data vector and a vector key value of each data vector sent by the aggregation node; the at least one data vector is stored to at least one data partition based on a vector key, wherein each data partition stores one data vector.
  2. 2. The system of claim 1, wherein the aggregation nodes in each cluster are further configured to: reading data vectors of all data partitions in all computing nodes in the cluster to obtain a plurality of data vectors; and combining the plurality of data vectors based on vector key values to obtain a local vector.
  3. 3. The system of claim 1, wherein the aggregation nodes in each cluster are further configured to: Transmitting the data vector to aggregation nodes of other clusters; Or alternatively And receiving data vectors sent by aggregation nodes of other clusters.
  4. 4. The system of claim 1, wherein the computing nodes in each cluster are further configured to: And carrying out encryption processing and decryption processing on the data vector according to the stored secret key to obtain the processed data vector.
  5. 5. The system of any one of claims 1 to 4, wherein the distributed vector processing system is configured to process data vectors in a federated learning scenario.
  6. 6. A method of distributed vector processing, characterized in that it is applied to an aggregation node of any cluster in the distributed vector processing system according to any one of claims 1 to 5, the method comprising: Acquiring a calculation instruction, wherein the calculation instruction is used for indicating to calculate the data vector in the cluster and comprises an operation type and a data identifier; Determining an operation instruction according to the operation type and the data identifier, wherein the operation instruction comprises an operator and a vector key value; and respectively sending the operation instruction to each computing node in the cluster.
  7. 7. The method of claim 6, wherein if the operation type is a binary operation, the operation instruction further comprises a left value variable and a right value variable.
  8. 8. The method according to claim 6 or 7, characterized in that the method further comprises: splitting the received data to be stored according to the number of the data partitions in the cluster to obtain a plurality of data vectors; Determining a vector key value corresponding to each data vector according to a preset rule; And according to the number of the data partitions on each computing node, respectively sending the plurality of data vectors and the vector key values to each computing node in the cluster.
  9. 9. The method of claim 8, wherein the method further comprises: reading data vectors of all data partitions in all computing nodes in the cluster to obtain a plurality of data vectors; and combining the plurality of data vectors based on vector key values to obtain a local vector.
  10. 10. The method according to claim 6 or 7, characterized in that the method further comprises: Transmitting the data vector to aggregation nodes of other clusters; Or alternatively And receiving data vectors sent by aggregation nodes of other clusters.
  11. 11. The method according to claim 6 or 7, wherein the data vector is a data vector in a bang-study scenario.
  12. 12. A method of distributed vector processing, applied to a compute node of any cluster in a distributed vector processing system according to any one of claims 1 to 5, the method comprising: receiving an operation instruction sent by an aggregation node of the cluster, wherein the operation instruction comprises an operator and a vector key value; and according to the operator, carrying out operation on the data vector corresponding to the vector key value in at least one local partition to obtain the operated data vector.
  13. 13. The method of claim 12, wherein if the operation type is a binary operation, the operation instruction further comprises a left value variable and a right value variable; Correspondingly, according to the operator, the operation is performed on the data vector corresponding to the vector key value in at least one local partition, so as to obtain the operated data vector, which comprises the following steps: And according to the operator, the left value variable and the right value variable operate on the data vector corresponding to the vector key value in at least one local partition to obtain the operated data vector.
  14. 14. The method according to claim 12 or 13, characterized in that the method further comprises: receiving at least one data vector and a vector key value of each data vector sent by the aggregation node; the at least one data vector is stored to at least one data partition based on a vector key, wherein each data partition stores one data vector.
  15. 15. The method according to claim 12 or 13, wherein the computing, according to the operator, a data vector corresponding to the vector key in the local at least one partition, and before obtaining the computed data vector, the method further comprises: and if the data vector corresponding to the vector key value in the local at least one partition is an encryption vector, decrypting the data vector corresponding to the vector key value in the local at least one partition by adopting a pre-stored key to obtain a decrypted data vector.
  16. 16. The method of claim 15, wherein the method further comprises: and encrypting the operated data vector by adopting a pre-stored key.
  17. 17. The method according to claim 12 or 13, wherein the data vector is a data vector in a bang-study scenario.
  18. 18. A distributed vector processing apparatus, characterized in that an aggregation node applied to any cluster in the distributed vector processing system according to any one of claims 1 to 5, comprises: the receiving module is used for acquiring a calculation instruction, wherein the calculation instruction is used for indicating to calculate the data vector in the cluster and comprises an operation type and a data identifier; the processing module is used for determining an operation instruction according to the operation type and the data identification, wherein the operation instruction comprises an operator and a vector key value; and the sending module is used for respectively sending the operation instruction to each computing node in the cluster.
  19. 19. A distributed vector processing apparatus, characterized in that a computing node applied to any cluster in the distributed vector processing system of any one of claims 1 to 5, comprises: The receiving module is used for receiving an operation instruction sent by the aggregation node of the cluster, wherein the operation instruction comprises an operator and a vector key value; and the processing module is used for carrying out operation on the data vector corresponding to the vector key value in at least one local partition according to the operator to obtain the operated data vector.
  20. 20. An electronic device comprising a memory, a processor and an interactive interface, the memory having stored thereon a computer program executable on the processor, the computer program implementing the steps of the distributed vector processing method according to any of claims 6 to 17 when executed by the processor.

Description

Distributed vector processing method, device, system and medium Technical Field The present invention relates to the field of computer technologies, and in particular, to a distributed vector processing method, device, system, and medium. Background With the continuous development of computer technology, federal learning (also called federal machine learning) is also widely used in various technical fields, and federal learning is a machine learning framework which can effectively help a plurality of institutions to perform data use and machine learning modeling under the condition of meeting the requirements of user privacy protection, data security and government regulations. In the current federal learning field, to develop a federal learning algorithm, the development implementation mode of a single machine is different from the distributed development implementation mode of multiple machines. The programming of the single-machine version is simple and visual, the participants can better organize and process complete data of all parties, and the participants can refer to the algorithm realization of the existing non-encryption scene to finish the development of the federal algorithm, and the disadvantage is that only a small amount of data can be processed. However, the standalone version developed by the current developers cannot be directly migrated into the distributed version. In summary, there is no simple and convenient distributed development architecture in the prior art, and a processing scheme capable of solving the problem of large data volume in federal learning. Disclosure of Invention The invention mainly aims to provide a distributed vector processing method, a device, a system and a medium, and provides a simple and convenient distributed development system, which can solve the problem of processing a large amount of data in a federal learning scene and can be directly migrated to a distributed version for application through development and application of the system. To achieve the above object, the present invention provides a distributed vector processing system, including: The system comprises at least two clusters, wherein each cluster comprises a plurality of nodes, and the number of the nodes in each cluster is the same; in each cluster, the plurality of nodes comprise two nodes, namely an aggregation node and a calculation node; the aggregation nodes in each cluster are used for managing data in the cluster where the aggregation nodes are located, and the calculation nodes are used for storing and calculating the data of at least one partition. In one embodiment, the aggregation nodes in each cluster are also used to execute and/or distribute the compute commands. In one embodiment, the aggregation nodes in each cluster are also used to communicate with other clusters. In one embodiment, the aggregation nodes in each cluster are specifically configured to: Acquiring a calculation instruction, wherein the calculation instruction is used for indicating to calculate the data vector in the cluster and comprises an operation type and a data identifier; Determining an operation instruction according to the operation type and the data identifier, wherein the operation instruction comprises an operator and a vector key value; sending the operation instruction to each computing node in the cluster respectively; accordingly, the computing nodes in each cluster are specifically configured to: receiving an operation instruction sent by an aggregation node of the cluster, wherein the operation instruction comprises an operator and a vector key value; and according to the operator, carrying out operation on the data vector corresponding to the vector key value in at least one local partition to obtain the operated data vector. In a specific embodiment, the aggregation nodes in each cluster are further configured to: splitting the received data to be stored according to the number of the data partitions in the cluster to obtain a plurality of data vectors; Determining a vector key value corresponding to each data vector according to a preset rule; According to the number of the data partitions on each computing node, the plurality of data vectors and the vector key values are respectively sent to each computing node in the cluster; accordingly, the computing nodes in each cluster are also to: receiving at least one data vector and a vector key value of each data vector sent by the aggregation node; the at least one data vector is stored to at least one data partition based on a vector key, wherein each data partition stores one data vector. In a specific embodiment, the aggregation nodes in each cluster are further configured to: reading data vectors of all data partitions in all computing nodes in the cluster to obtain a plurality of data vectors; and combining the plurality of data vectors based on vector key values to obtain a local vector. In a specific embodiment, the aggregation nodes in