CN-122018814-A - Scheduling processing method, system, medium and product of distributed network disk

CN122018814ACN 122018814 ACN122018814 ACN 122018814ACN-122018814-A

Abstract

A dispatching processing method, a dispatching processing system, a dispatching processing medium and a dispatching processing product for a distributed network disk relate to the technical field of distributed storage. By implementing the method, the scheduling processing system maps the unique identifier of the data block to be transmitted to the virtual hash ring through the consistent hash algorithm, and accurately marks the preset number of candidate nodes, so that the quick distribution characteristic of the hash mapping is reserved, and the limitation of single node binding is avoided. The scheduling processing system acquires real-time states such as disk queue depth, available bandwidth and the like of the candidate nodes through the link detection packet, calculates a real-time load resistance value by combining network round trip delay, simultaneously predicts data transmission time consumption and queuing waiting time consumption, and finally screens out a target execution node with the shortest expected task completion time. The closed loop design of hash positioning, real-time state evaluation and comprehensive time-consuming sequencing realizes the dynamic matching of the data access request and the performance of the storage nodes, and effectively avoids the condition that part of nodes are overloaded and congested and part of nodes are idle in resource waste.

Inventors

WEI HAOYANG
LIU KAIXUAN
YANG BO
YANG JINGXUE
CHEN XIAOJIAN
LIN JIANXIONG

Assignees

福建国科信息科技有限公司

Dates

Publication Date: 20260512
Application Date: 20260326

Claims (8)

1. A method for scheduling a distributed network disk, applied to a scheduling processing system, the method comprising: Receiving a data access request sent by a client, wherein the data access request comprises a unique identifier of a data block to be transmitted and the size of data volume; Calculating the unique identifier based on a consistent hash algorithm to obtain a target hash value, and mapping the target hash value onto a virtual hash ring, wherein the virtual hash ring is used for representing logic positions of a plurality of storage nodes in a hash space; traversing the virtual hash ring in a clockwise direction by taking the corresponding position of the target hash value on the virtual hash ring as a starting point, and selecting a preset number of storage nodes traversed in sequence as candidate nodes; respectively sending link detection packets marked with sending time stamps to the candidate nodes, receiving state response packets which are fed back by the candidate nodes and comprise disk queue depth and available bandwidth, and recording receiving time stamps of the state response packets; Calculating the network round trip delay of the candidate node according to the difference value of the sending timestamp and the receiving timestamp, calculating the real-time load resistance value of the candidate node based on the depth of the disk queue and the network round trip delay, and mapping the real-time load resistance value into corresponding queuing time consumption; estimating data transmission time consumption by combining the data size and the available bandwidth of the candidate node, and adding the data transmission time consumption and the queuing time consumption to obtain the expected task completion time length of the candidate node; Comparing the expected task completion time of the candidate nodes, screening out the candidate node with the minimum expected task completion time as a target execution node, generating a scheduling instruction comprising a target network address corresponding to the target execution node, and sending the scheduling instruction to the client to instruct the client to establish data transmission connection with the target execution node; The method comprises the steps of determining the ratio of the network round trip delay to the preset reference network delay as a transmission impedance index, determining the ratio of the disk queue depth to the maximum concurrent I/O threshold of the candidate node as a first I/O saturation, carrying out exponential weighted amplification on the first I/O saturation to obtain a second I/O saturation, and carrying out linear summation on the transmission impedance index and the second I/O saturation to obtain the real-time load resistance value of the candidate node; The method comprises the steps of mapping the real-time load resistance value into corresponding queuing time consumption, and specifically comprises the steps of obtaining a storage medium type of a candidate node, determining a unit resistance time consumption coefficient corresponding to the storage medium type, wherein the unit resistance time consumption coefficient is used for representing average time required by processing data corresponding to the storage medium type under unit load resistance, multiplying the real-time load resistance value by the unit resistance time consumption coefficient to obtain basic queuing time consumption, obtaining an average I/O response jitter value of the candidate node in a preset historical time window, and correcting the basic queuing time consumption based on the average I/O response jitter value to obtain the queuing time consumption.
2. The method according to claim 1, wherein the estimating the data transmission time consumption by combining the data size and the available bandwidth of the candidate node comprises: judging whether the data size is smaller than a preset TCP slow start threshold value or not; If yes, calculating a bandwidth utilization coefficient according to the data size, wherein the bandwidth utilization coefficient is positively correlated with the data size; correcting the available bandwidth by utilizing the bandwidth utilization coefficient to obtain an effective transmission bandwidth; and calculating the ratio of the data size to the effective transmission bandwidth to obtain the time consumption of data transmission.
3. The method according to claim 2, wherein before the step of transmitting the link probing packets respectively marked with the transmission time stamps to the candidate nodes, the method further comprises: Inquiring whether a history state record of the candidate node exists in a local cache; If the history state record of the candidate node exists in the local cache and the update time of the history state record is within a preset effective time window, directly reading the depth of a disk queue and the available bandwidth in the history state record; and if the history state record of the candidate node does not exist in the local cache and/or the update time of the history state record is outside a preset effective time window, executing the step of respectively sending the link detection packets marked with the sending time stamp to the candidate node.
4. The method of claim 1, wherein after the step of traversing the virtual hash ring in a clockwise direction starting from a corresponding location of the target hash value on the virtual hash ring, selecting a predetermined number of storage nodes traversed in sequence as candidate nodes, the method further comprises: judging whether the unique identifier is matched with a target data identifier in a hot spot data list, wherein the hot spot data list comprises a plurality of data identifiers, and the target data identifier is any one data identifier; if yes, inquiring a target node corresponding to the target data identifier; judging whether an intersection node exists between the target node and the candidate node; if yes, reducing the disk queue depth of the intersection node according to a preset cache hit weight so as to reduce the real-time load resistance value of the intersection node.
5. The method according to claim 1, wherein the method further comprises: acquiring the total sending count of the link detection packets sent to the candidate nodes within a preset time length and the total receiving count of the corresponding state response packets successfully received; calculating a difference value between the total transmission count and the total reception count, and determining a ratio of the difference value to the total transmission count as a data packet loss rate; calculating a bandwidth breakage coefficient according to the data packet loss rate, wherein the higher the data packet loss rate is, the larger the bandwidth breakage coefficient is; the available bandwidth of the candidate node is reduced by utilizing the bandwidth reduction coefficient, and the actual effective bandwidth of the candidate node is obtained; Dividing the data size by the actual effective bandwidth, and calculating to obtain the time consumption of data transmission.
6. A dispatch processing system comprising one or more processors and memory coupled to the one or more processors, the memory to store computer program code comprising computer instructions that the one or more processors invoke to cause the dispatch processing system to perform the method of any of claims 1 to 5.
7. A computer readable storage medium comprising instructions which, when run on a dispatch processing system, cause the dispatch processing system to perform the method of any one of claims 1 to 5.
8. A computer program product, characterized in that the computer program product, when run on a dispatch processing system, causes the dispatch processing system to perform the method of any one of claims 1 to 5.

Description

Scheduling processing method, system, medium and product of distributed network disk Technical Field The application relates to the technical field of distributed storage, in particular to a dispatching processing method, a dispatching processing system, a dispatching processing medium and a dispatching processing product of a distributed network disk. Background With the rapid development of internet technology and the continuous growth of data scale, distributed network disks are widely used. The distributed network disk can effectively improve the storage capacity and the data access efficiency of the system by dispersedly storing the data on a plurality of storage nodes. In the distributed network disk, how to reasonably schedule the data access request, and realize the efficient transmission and load balancing of the data, becomes an important technical subject. At present, a distributed network disk system adopts a fixed hash mapping mode to schedule data. Specifically, the distributed network disk system calculates a hash value according to the identification information of the data block, and maps the hash value directly into a pre-divided hash space. Each storage node is allocated with a fixed hash value interval, and when a data access request is received, the distributed network disk system allocates the data access request to the storage node corresponding to the interval to which the hash value belongs for processing. This way a fast scheduling distribution of data access requests is achieved. However, in practical applications, there are often large differences in hardware performance, network status, and traffic load of storage nodes. The scheduling manner of the fixed hash map does not consider the real-time state of the storage nodes, which may cause part of storage nodes with poor performance or heavy load to continuously receive the data access request, while other storage nodes are in a relatively idle state. When a large number of data access requests exist, the problem of unbalanced load can cause serious access congestion of certain storage nodes, and the data transmission efficiency is affected. Disclosure of Invention The application provides a dispatching processing method, a dispatching processing system, a dispatching processing medium and a dispatching processing product for a distributed network disk, which are used for improving the overall data transmission efficiency, reducing the average task completion time delay and simultaneously guaranteeing the load balance of the distributed network disk system. In a first aspect, the present application provides a method for scheduling a distributed network disk, which is applied to a scheduling processing system, and the method includes receiving a data access request sent by a client, where the data access request includes a unique identifier of a data block to be transmitted and a data size; calculating the unique identifier based on a consistent hash algorithm to obtain a target hash value, and mapping the target hash value onto a virtual hash ring, wherein the virtual hash ring is used for representing logic positions of a plurality of storage nodes in a hash space; traversing the virtual hash ring along the clockwise direction by taking the corresponding position of the target hash value on the virtual hash ring as a starting point, selecting a preset number of storage nodes which are traversed in sequence as candidate nodes, respectively sending link detection packets marked with sending time stamps to the candidate nodes, receiving state response packets which are fed back by the candidate nodes and comprise disk queue depths and available bandwidths, recording the receiving time stamps of the state response packets, calculating the network round trip delay of the candidate nodes according to the difference value of the sending time stamps and the receiving time stamps, calculating the real-time load resistance value of the candidate nodes based on the disk queue depths and the network round trip delay, mapping the real-time load resistance value to the corresponding queuing time, estimating the data transmission time consumption by combining the data volume size and the available bandwidths of the candidate nodes, adding the data transmission time consumption and the queuing time consumption to obtain the expected task completion time length of the candidate nodes, comparing the expected task completion time length of the candidate nodes, screening out the candidate nodes with the minimum expected task completion time length as target execution nodes, generating a scheduling instruction comprising the target network address corresponding to the target execution nodes, and sending the scheduling instruction to a client, to instruct the client to establish a data transfer connection with the target execution node. By adopting the technical scheme, firstly, the scheduling processing system maps the unique identifier of the data