CN-116032926-B - Ignite-based non-centralized rack sensing method, system, electronic equipment and computer-readable storage medium

CN116032926BCN 116032926 BCN116032926 BCN 116032926BCN-116032926-B

Abstract

The invention provides a method, a system, equipment and a medium for sensing a non-centralized rack based on ignite in the technical field of data distributed access, which comprises the following steps of calculating weights of all partitions at all nodes through a highest random weight algorithm, sequencing all nodes based on the weights to obtain a node weight sequence, setting the partition held by the node with the highest weight as a main partition, setting the node holding the main partition as a main node, judging the network distance between the rest of all nodes and the main node based on the topological relation between the main computer and the node, setting a quantity threshold value for identifying the allowed holding of all racks, distributing the auxiliary partitions based on the quantity threshold value and the network distance, and distributing the rest of the auxiliary partitions based on the node weight sequence, wherein the step S20 is used for judging the network distance between the rest of all the nodes and the main node based on the topological relation between the main computer and the nodes. The invention has the advantage of greatly improving the reliability and efficiency of data access.

Inventors

LV CHONGYI

Assignees

福建新大陆软件工程有限公司

Dates

Publication Date: 20260505
Application Date: 20230110

Claims (4)

1. A ignite-based non-centralized rack sensing method is characterized by comprising the following steps: Step S10, calculating the weight of each partition in each node through a highest random weight algorithm, and sequencing each node based on the weight to obtain a node weight sequence, wherein the nodes are deployed in a host, and the host is deployed in a rack; Step S20, setting a partition held by the node with the highest weight as a main partition, and setting a node holding the main partition as a main node; Step S30, setting the network distance of the nodes which are in the same host as the main node as a neighbor, and setting the network distance of each node which is not the neighbor as a far neighbor; step S40, setting a quantity threshold value for identifying the allowed holding of the secondary partitions by each rack, and distributing the secondary partitions based on the quantity threshold value and the network distance; step S50, distributing the rest secondary partitions based on the node weight sequence; The step S40 specifically includes: Step S41, setting a threshold value for identifying the number of allowed auxiliary partitions of each rack, judging whether the network distance of the next node is far or not based on the node weight sequence, if not, not distributing the auxiliary partitions to the node to judge the next node, if so, entering step S42; step S42, judging whether the number of the sub-partitions already held by the rack corresponding to the current node is smaller than the number threshold, if so, distributing the sub-partitions to the current node, and judging the next node; step S43, judging whether the allocation of all the secondary partitions is finished, if not, proceeding to step S50, and if so, ending the flow.
2. Ignite-based non-centralized rack sensing system is characterized by comprising the following modules: The node weight calculation module is used for calculating the weight of each partition in each node through the highest random weight algorithm, and sequencing each node based on the weight to obtain a node weight sequence; the main partition allocation module is used for setting a partition held by the node with the highest weight as a main partition and setting a node holding the main partition as a main node; The network distance calculation module is used for setting the network distance of the nodes which are in the same host with the main node as a neighbor and setting the network distance of each node which is not the neighbor as a far neighbor; The first auxiliary partition allocation module is used for setting a quantity threshold value for identifying the allowed holding auxiliary partitions of each rack and allocating the auxiliary partitions based on the quantity threshold value and the network distance; a second sub-partition allocation module, configured to allocate the remaining sub-partitions based on the node weight order; the first secondary partition allocation module specifically includes: the network distance checking unit is used for setting a quantity threshold value for identifying the allowed holding secondary partitions of each rack, judging whether the network distance of the next node is far or not based on the node weight sequence, if not, not distributing the secondary partitions to the node to judge the next node, and if so, entering the quantity threshold value checking unit; The system comprises a quantity threshold value checking unit, a next node judgment unit and a frame judgment unit, wherein the quantity threshold value checking unit is used for judging whether the quantity of the auxiliary partitions which are already held by the frame corresponding to the current node is smaller than the quantity threshold value, if so, the auxiliary partitions are distributed to the current node, and the next node judgment is carried out; the auxiliary partition allocation checking unit is used for judging whether the allocation of all the auxiliary partitions is finished, if not, entering a second auxiliary partition allocation module, and if so, ending the flow.
3. Ignite-based, non-centralized, rack-aware electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of claim 1 when executing the program.
4. Ignite-based, non-centralized, rack-aware computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of claim 1.

Description

Ignite-based non-centralized rack sensing method, system, electronic equipment and computer-readable storage medium Technical Field The invention relates to the technical field of data distributed access, in particular to a ignite-based non-centralized rack sensing method, a ignite-based non-centralized rack sensing system, electronic equipment and a computer-readable storage medium. Background Ignite is a memory-based distributed key value storage, which can be regarded as a distributed partitioning hash, in the partitioning mode, each node under the host has a fixed number of partitions, and obtains which partition the data is to be stored into by modulo the hash value of the data to be stored and the partition number, the same partition number is divided into a main partition and a sub partition on different nodes, the number of the main partition is only one, the number of the sub partition can be multiple, the main partition is referred to as main data for the same data, and the sub partition is referred to as copy data, as shown in fig. 5. The main partition and the auxiliary partition are distributed to which node of which host is realized by a Highest Random Weight (HRW) algorithm, namely, the hash value of the combination of the partition number and the information of each node is used as a parameter, the partition number and the weight of each node are calculated by the highest random weight algorithm, the nodes are ordered from high to low according to the weight, the node with the highest weight holds the main partition, and other nodes preferentially hold the auxiliary partition. However, the existing partition distribution technology is based on a host, and aims to distribute a main partition and a secondary partition to nodes of each host, and belongs to partition distribution of host dimensions, and the existing partition distribution technology has the following defects that 1, the situation that the hosts of the main partition and the secondary partition are located in the same rack can occur, when a network or a power supply of the rack is disconnected, all nodes in the rack can be disconnected, partition loss can occur, and data loss is further caused, 2, data of the main partition and the secondary partition can be distributed in the same or a limited number of racks, when nodes located in other racks access the data, the nodes need to cross the host and the racks, so that the network distance is slightly larger, and further the data access efficiency is reduced. Therefore, how to provide a method, a system, an electronic device and a computer readable storage medium for sensing a non-centralized rack based on ignite, so as to improve the reliability and efficiency of data access, is a technical problem to be solved. Disclosure of Invention The invention aims to solve the technical problem of providing ignite-based non-centralized rack sensing method, system, electronic equipment and computer-readable storage medium, so as to improve the reliability and efficiency of data access. In a first aspect, the present invention provides a method for sensing a non-centralized rack based on ignite, comprising the steps of: Step S10, calculating the weight of each partition in each node through a highest random weight algorithm, and sequencing each node based on the weight to obtain a node weight sequence; Step S20, setting a partition held by the node with the highest weight as a main partition, and setting a node holding the main partition as a main node; step S30, judging the network distance between the rest nodes and the main node based on the topological relation between the host and the nodes; step S40, setting a quantity threshold value for identifying the allowed holding of the secondary partitions by each rack, and distributing the secondary partitions based on the quantity threshold value and the network distance; And step S50, distributing the rest secondary partitions based on the node weight sequence. Further, in the step S10, the node is disposed in a host, and the host is disposed in a rack. Further, the step S30 specifically includes: The network distance of the nodes which are in the same host with the main node is set as a neighbor, and the network distance of each node which is not the neighbor is set as a far neighbor. Further, the step S40 specifically includes: Step S41, setting a threshold value for identifying the number of allowed auxiliary partitions of each rack, judging whether the network distance of the next node is far or not based on the node weight sequence, if not, not distributing the auxiliary partitions to the node to judge the next node, if so, entering step S42; step S42, judging whether the number of the sub-partitions already held by the rack corresponding to the current node is smaller than the number threshold, if so, distributing the sub-partitions to the current node, and judging the next node; step S43, judging whether the allocation of all the second