CN-115357398-B - Data transmission method, device, equipment and storage medium

CN115357398BCN 115357398 BCN115357398 BCN 115357398BCN-115357398-B

Abstract

The application provides a data transmission method, a device, equipment and a storage medium, wherein the method comprises the following steps: the first system obtains the data transmission time length of the second system for transmitting the current batch of data to each of the plurality of first processing nodes, determines the current load of each first processing node according to the memory overflow index, the data transmission time length and the plurality of resource performance parameters of each first processing node, determines the target processing node from the plurality of first processing nodes according to the current load of each first processing node, and finally sends the identification of the target processing node to the second system so as to enable the target processing node to receive the data transmitted by the second system in the next batch. In the scheme, the processing capacity and the state of different processing nodes in the first system are fully considered, so that the load balance is improved.

Inventors

XIONG LINBO

Assignees

深圳前海微众银行股份有限公司
深圳前海微众银行股份有限公司

Dates

Publication Date: 20260421
Application Date: 20220905
Priority Date: 20220905

Claims (11)

1. A data transmission method applied to a first system, the first system comprising a plurality of processing nodes, the method comprising: acquiring data transmission time length of a second system for transmitting current batch data to each of a plurality of first processing nodes, wherein the plurality of first processing nodes are determined from the plurality of processing nodes; Determining the current load of each first processing node according to the memory overflow index, the data transmission duration and the plurality of resource performance parameters of each first processing node, wherein the memory overflow index of each first processing node is used for indicating whether the memory of the first processing node overflows or not; determining a target processing node from the plurality of first processing nodes according to the current load of each first processing node; Transmitting the identification of the target processing node to the second system so that the data transmitted by the next batch of the second system is received through the target processing node; after determining the current load of each first processing node according to the memory overflow index, the data transmission duration and the plurality of resource performance parameters of each first processing node, the method further comprises: Determining a current load of each first processing node as a current integrated load sample exponentially weighted moving average SLEWMA value for the first processing node; Correspondingly, the determining the target processing node from the plurality of first processing nodes according to the current load of each first processing node includes: Determining the target processing node meeting a preset condition from the plurality of first processing nodes according to the current SLEWMA value of each first processing node, wherein the preset condition is that the current SLEWMA value of the first processing node is larger than a preset SLEWMA value or the current SLEWMA value is the largest preset number of first processing nodes; After determining the target processing node meeting a preset condition from the plurality of first processing nodes according to the current SLEWMA value of each first processing node, the method further includes: according to the current SLEWMA value of the target processing node, placing the identification of the target processing node into an initial empty large top heap to generate a first large top heap; correspondingly, the sending the identification of the target processing node to the second system includes: sequentially sending the identification of the target processing node in the first large top heap to the second system according to the sequence from bottom to top; After said sending the identification of the target processing node to the second system, the method further comprises: determining the current load of each first processing node as a historical load, and re-determining a plurality of second processing nodes from the plurality of processing nodes; Determining the current load of each second processing node according to the memory overflow index, the plurality of resource performance parameters and the data transmission duration of each second processing node; Redetermining a new target processing node from the plurality of second processing nodes and the target processing node according to the current load of each second processing node and the historical load of the target processing node; transmitting the identification of the new target processing node to the second system; after determining the current load of each second processing node according to the memory overflow index, the plurality of resource performance parameters and the data transmission duration of each second processing node, the method further comprises: if any second processing node does not have the history load, determining the current load of the second processing node as the current SLEWMA value of the second processing node; if any second processing node has the history load, calculating a current SLEWMA value of the second processing node according to the current load of the second processing node, the history load and a preset burr variable, wherein the burr variable is determined according to the history load; Determining a history SLEWMA value of the target processing node according to the history load of the target processing node and the burr variable; Correspondingly, the redefining a new target processing node from the plurality of second processing nodes and the target processing node according to the current load of each second processing node and the historical load of the target processing node comprises the following steps: and re-determining the new target processing node from a plurality of second processing nodes and the target processing node according to the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node.
2. The method of claim 1, wherein determining the current load of each first processing node based on the memory overflow indicator, the data transmission duration, and the plurality of resource performance parameters of each first processing node comprises: Determining a weight value of each resource performance according to the memory overflow index of each first processing node and a plurality of resource performance parameters, wherein the weight value of each resource performance is used for representing the importance of the resource performance to the current load of the first processing node; And determining the current load of each first processing node according to the weight value of each resource performance, the data transmission duration of each first processing node and a plurality of resource performance parameters.
3. The method of claim 2, wherein determining the weight value for each resource performance based on the memory overflow indicator and the plurality of resource performance parameters for each first processing node comprises: Determining a plurality of initial resource performance parameters according to the resource performance parameters of each first processing node aiming at each resource performance, wherein each initial resource performance parameter is in a parameter range of the resource performance parameters of the plurality of first processing nodes; determining a base index of each initial resource performance parameter according to the memory overflow index of each first processing node and the resource performance parameter; and determining the weight value of the resource performance according to the minimum value of the base index.
4. A method according to claim 3, wherein said determining, for each resource performance, a plurality of initial resource performance parameters from the resource performance parameter for each first processing node comprises: Sequencing the resource performance parameters of the plurality of first processing nodes according to the sequence from big to small for each resource performance to obtain a sequencing result; and determining the average value of two adjacent resource performance parameters as an initial resource performance parameter according to the sorting result.
5. The method of claim 1, wherein after the new target processing node is re-determined from the plurality of second processing nodes and the target processing node based on the historical SLEWMA values of the target processing node and the current SLEWMA value of each second processing node, the method further comprises: Updating the first large top heap according to the current SLEWMA value and the identifier of the new target processing node to generate a second large top heap; correspondingly, the sending the identification of the new target processing node to the second system includes: and sequentially sending the identification of the new target processing node in the second large top heap to the second system according to the sequence from bottom to top.
6. The method according to any one of claims 1 to 5, wherein determining the current load of each first processing node according to the weight value of each resource performance, the data transmission duration of each first processing node, and the plurality of resource performance parameters comprises: And determining the current load of each first processing node according to the weight value of each resource performance, the data transmission time length of each first processing node, the preset weight value and a plurality of resource performance parameters.
7. The method of any one of claims 1 to 4, wherein the plurality of resource performance parameters for each first processing node comprises at least two of a current traffic of the first processing node, a CPU usage, a memory usage, an I/O usage of the network card, and an I/O usage of the network card of the disk.
8. The method of any one of claims 1 to 4, wherein prior to the acquiring a data transmission duration for the second system to transmit the current batch of data to each of the plurality of first processing nodes, the method further comprises: And sending the identification of each first processing node, the target data type of the current batch data, the identification of the target port of the second system and the identification of the target process running in the second system to the second system.
9. A data transmission apparatus for use in a first system, the first system comprising a plurality of processing nodes, the apparatus comprising: an acquisition module, configured to acquire a data transmission duration of transmitting current batch data by a second system to each of a plurality of first processing nodes, where the plurality of first processing nodes are determined from the plurality of processing nodes; the processing module is used for determining the current load of each first processing node according to the memory overflow index, the data transmission duration and the plurality of resource performance parameters of each first processing node, wherein the memory overflow index of each first processing node is used for indicating whether the memory of the first processing node overflows or not; The processing module is further used for determining a target processing node from the plurality of first processing nodes according to the current load of each first processing node; The sending module is used for sending the identification of the target processing node to the second system so as to enable the target processing node to receive data transmitted by the second system in the next batch; After determining the current load of each first processing node according to the memory overflow index, the data transmission duration and the plurality of resource performance parameters of each first processing node, the processing module is further configured to: determining a current load of each first processing node as a current SLEWMA value of the first processing node; the processing module is specifically configured to: Determining target processing nodes meeting a preset condition from a plurality of first processing nodes according to the current SLEWMA value of each first processing node, wherein the preset condition is that the current SLEWMA value of the first processing nodes is larger than a preset SLEWMA value or the current SLEWMA value is the largest preset number of first processing nodes; after determining a target processing node meeting a preset condition from a plurality of first processing nodes according to the current SLEWMA value of each first processing node, the processing module is further configured to put the identifier of the target processing node into an initial empty large top heap according to the current SLEWMA value of the target processing node, so as to generate a first large top heap; the sending module is specifically configured to: according to the sequence from bottom to top, the identification of the target processing node in the first large top heap is sequentially sent to a second system; After the identification of the target processing node is sent to the second system, the processing module is further configured to determine a current load of each first processing node as a historical load, and re-determine a plurality of second processing nodes from the plurality of processing nodes; The processing module is further configured to determine a current load of each second processing node according to the memory overflow index, the plurality of resource performance parameters, and the data transmission duration of each second processing node; The processing module is further used for redefining a new target processing node from the plurality of second processing nodes and the target processing node according to the current load of each second processing node and the historical load of the target processing node; the sending module is further configured to send an identifier of the new target processing node to the second system; After determining the current load of each second processing node according to the memory overflow index, the plurality of resource performance parameters and the data transmission duration of each second processing node, the processing module is further configured to: if no history load exists in any second processing node, determining the current load of the second processing node as the current SLEWMA value of the second processing node; if any second processing node has a history load, calculating a current SLEWMA value of the second processing node according to the current load of the second processing node, the history load and a preset burr variable, wherein the burr variable is determined according to the history load; Determining a historical SLEWMA value of the target processing node according to the historical load of the target processing node and the burr variable; the processing module is specifically configured to: a new target processing node is re-determined from the plurality of second processing nodes and the target processing node based on the historical SLEWMA values of the target processing node and the current SLEWMA value of each second processing node.
10. An electronic device comprising a processor, a transceiver, a memory and computer program instructions stored on the memory and executable on the processor, wherein the processor is configured to implement the data transmission method of any one of claims 1 to 8 when executing the computer program instructions.
11. A computer-readable storage medium, wherein computer-executable instructions are stored in the computer-readable storage medium, which when executed by a processor are adapted to implement the data transmission method of any one of claims 1 to 8.

Description

Data transmission method, device, equipment and storage medium Technical Field The present application relates to the technical field of financial science and technology, and in particular, to a data transmission method, apparatus, device and storage medium. Background With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to the financial technology, and the data transmission technology is not exceptional. However, as the financial business data present an explosive growth trend nowadays, and meanwhile, the data transmission technology is more highly required based on the aspects of safety, real-time performance, load balancing and the like. At present, the financial service data is generally transmitted by accessing a financial service system to a third party processing system (such as a log system), and transmitting the data to the third party processing system through a polling algorithm or a weighted polling algorithm by a related interface, so that each processing node in the third party processing system performs load balancing on the data to land. However, in the prior art, because the capacities of the processing nodes in the third party processing system are different, and the data volume sent by the financial service system each time is different according to the polling algorithm or the weighted polling algorithm, there is a problem of poor load balancing. Disclosure of Invention The application provides a data transmission method, a device, equipment and a storage medium, which are used for solving the problem of unbalanced load in the prior art. In a first aspect, an embodiment of the present application provides a data transmission method, which is applied to a first system, where the first system includes a plurality of processing nodes, and the method includes: A data transmission duration of the second system transmitting the current batch of data to each of a plurality of first processing nodes determined from the plurality of processing nodes is obtained. And determining the current load of each first processing node according to the memory overflow index, the data transmission duration and the plurality of resource performance parameters of each first processing node, wherein the memory overflow index of each first processing node is used for indicating whether the memory of the first processing node overflows. And determining a target processing node from the plurality of first processing nodes according to the current load of each first processing node. And sending the identification of the target processing node to the second system so that the data transmitted by the next batch of the second system is received through the target processing node. In one possible design of the first aspect, the determining the current load of each first processing node according to the memory overflow index, the data transmission duration, and the plurality of resource performance parameters of each first processing node includes: And determining a weight value of each resource performance according to the memory overflow index of each first processing node and a plurality of resource performance parameters, wherein the weight value of each resource performance is used for indicating the importance of the resource performance to the current load of the first processing node. And determining the current load of each first processing node according to the weight value of each resource performance, the data transmission duration of each first processing node and a plurality of resource performance parameters. Optionally, the determining the weight value of each resource performance according to the memory overflow index of each first processing node and the plurality of resource performance parameters includes: For each resource performance, determining a plurality of initial resource performance parameters according to the resource performance parameters of each first processing node, wherein each initial resource performance parameter is in a parameter range of the resource performance parameters of the plurality of first processing nodes. And determining the base index of each initial resource performance parameter according to the memory overflow index of each first processing node and the resource performance parameter. And determining the weight value of the resource performance according to the minimum value of the base index. Optionally, for each resource performance, determining a plurality of initial resource performance parameters according to the resource performance parameter of each first processing node includes: And sequencing the resource performance parameters of the plurality of first processing nodes according to the sequence from big to small for each resource performance, and obtaining a sequencing result. And determining the average value of two adjacent resource perfor