Search

CN-121636263-B - Method and device for self-adaptive adjustment of copies based on graph database

CN121636263BCN 121636263 BCN121636263 BCN 121636263BCN-121636263-B

Abstract

The invention discloses a method and a device for self-adaptive adjustment of a copy based on a graph database. The method comprises the steps of setting a corresponding expected copy value for each pre-defined Zone in a cluster creation stage, detecting the copy state reported by a storage layer in real time to obtain PartView of current data copy distribution and StoreView of physical resource use conditions, comparing the copy number in PartView with the expected copy value, generating a copy change instance when the copy number is different from the expected copy value, selecting a target storage node through a filter and a speed limiter according to StoreView, executing the expansion or contraction operation of the copy, and ending the copy adjustment flow when the copy number is the same as the expected copy value. The invention can achieve the technical effect of improving the availability of the database.

Inventors

  • ZHANG JIE
  • YE XIAOMENG

Assignees

  • 杭州悦数科技有限公司

Dates

Publication Date
20260508
Application Date
20260205

Claims (9)

  1. 1. A method for graph database based replica adaptation, comprising: in the cluster creation stage, setting a corresponding expected copy value for each pre-defined Zone; detecting the copy state reported by the storage layer in real time to obtain PartView of current data copy distribution and StoreView of physical resource use condition; Comparing the number of copies in PartView to the desired copy value; When the number of the copies is different from the expected copy value, generating a copy change instance, selecting a target storage node through a filter and a speed limiter according to StoreView, and executing the capacity expansion or contraction operation of the copies; when the number of the copies is the same as the expected copy value, ending the copy adjustment flow; Wherein the method further comprises: when the main area fails, sending a metadata copy capacity-shrinking request to a metadata service metad's raft group; Executing member change operation on the raft group of the metadata service metad according to the metadata copy capacity-shrinking request, so that the metadata service metad of the backup area becomes a leader node; After the metadata service metad of the spare area becomes a leader node, sending a data copy capacity-shrinking request to the leader node; According to the data copy capacity shrinking request, performing capacity shrinking operation of the data copy on the leader node so that the leader of all data fragments is concentrated in the standby area, and providing writing service by the standby area; after the main area is restored, sending a metadata copy capacity expansion request to the metadata service metad, where the metadata copy capacity expansion request is used to restore the copy number of the metadata service metad; after the number of copies of the metadata service metad is recovered, sending a data copy capacity expansion request to a metadata service metad of the backup area of the leader node; Executing copy capacity expansion operation of the data fragments according to the data copy capacity expansion request and the expected copy value; and executing a balance leader operation under the condition that the copy number of all the data fragments reaches the expected copy value, enabling the leader of the data fragments to appear in the main area, and re-providing the writing service by the main area.
  2. 2. The method of graph database based replica adaptation of claim 1, wherein said selecting a target storage node through a filter according to said StoreView comprises: The filter selects a target storage node for carrying a new copy based on at least one factor including isolation, node capacity and status, number of snapshots, and copy filling efficiency.
  3. 3. The method for adaptive adjustment of a replica based on a graph database according to claim 2, wherein said performing a capacity expansion or contraction operation of the replica comprises: sending metadata copy adjustment instructions to the metadata service metad via the ngctl tool; sending data copy adjustment instructions to the metadata service metad via ngql means; the replica adjustment process is performed by the leader node.
  4. 4. The method for graph database based replica adaptation of claim 1, And maintaining each storage node through the speed limiter, wherein in the copy capacity expansion process, the allocation rate of selecting the storage nodes for a plurality of data fragments is controlled, so that the storage nodes are sequentially staggered and selected for uniform resource allocation when the storage nodes are selected for the plurality of data fragments.
  5. 5. The method of graph database based replica adaptation of claim 1, further comprising: When detecting that the disk failure of the storage node causes data loss, automatically shrinking the copy of the affected fragment; And when the number of the copies of the fragments is identified to be lower than the expected copy value in the subsequent detection, automatically triggering a copy capacity expansion process, selecting a new storage node, and issuing a data group to the new storage node to complete automatic recovery of data.
  6. 6. The method of graph database based replica adaptation of claim 1, further comprising: And constructing a single-copy cluster in a data import scene, wherein copy expansion is initiated by specifying the expected copy number of single Zones or multiple Zones, and the metadata service metad completes the distribution adjustment of the copies according to the filter and the speed limiter.
  7. 7. The method of graph database based replica adaptation of claim 1, further comprising: when the metadata service metad receives the copy change request, persisting change information to the local; And the background dispatching thread of the metadata service metad periodically reads global configuration, compares fragment information with the expected copy value, and triggers a copy adjustment flow when the fragment information is inconsistent with the expected copy value.
  8. 8. The method for graph database based replica adaptation of claim 1, wherein the method operates in a graph database system consisting of a metadata service metad, a storage service storaged, and a query service graphd, wherein the query service graphd periodically obtains a tile distribution view from the metadata service metad and adjusts access policies according to the availability status of Zone.
  9. 9. An apparatus for graph database based replica adaptation, comprising: the configuration module is used for setting a corresponding expected copy value for each pre-defined Zone in the cluster creation stage; The detection module is used for detecting the copy state reported by the storage layer in real time to obtain PartView of current data copy distribution and StoreView of physical resource use conditions; The comparison module is used for comparing the number of the copies in the PartView with the expected copy value; The first adjusting module is used for generating a copy change instance when the copy number is different from the expected copy value, selecting a target storage node through a filter and a speed limiter according to StoreView, and executing the capacity expansion or contraction operation of the copy; the second adjusting module is used for ending the copy adjusting flow when the copy number is the same as the expected copy value; When a main area fails, a metadata copy capacity reduction request is sent to a RAft group of a metadata service metad, member change operation is carried out on the RAft group of the metadata service metad according to the metadata copy capacity reduction request, the metadata service metad of a backup area becomes a leader node, after the metadata service metad of the backup area becomes the leader node, a data copy capacity reduction request is sent to the leader node, a data copy capacity reduction operation is carried out on the leader node according to the data copy capacity reduction request, so that the leader of all data fragments is concentrated in the backup area, write service is provided by the backup area, after the main area is restored, a metadata copy capacity reduction request is sent to the metadata copy capacity reduction service metad, the metadata copy capacity reduction request is used for restoring the copy number of the metadata service metad, after the copy number of the metadata copy capacity reduction service metad is restored, a metadata copy capacity metad of the backup area of the leader node is sent to the metadata copy service metad of the leader node, according to the data copy capacity reduction request is carried out, a data copy capacity reduction operation is carried out on the backup area according to the backup value, and a copy capacity reduction request is carried out on the backup area, and a copy capacity reduction value is carried out on the backup of the backup area.

Description

Method and device for self-adaptive adjustment of copies based on graph database Technical Field The invention relates to the technical application field of graph databases, in particular to a method and a device for adaptively adjusting copies based on a graph database. Background Conventional graph databases have some limitations in data storage and management, such as: regarding the static copy number, if the copy number is fixed during cluster deployment, the copy threshold cannot be lowered in time when the main area tidies up faults so as to ensure writable service; Regarding manual intervention, in the aspect of main and standby switching, manual operation and maintenance operations are needed for copy completion and the like, and RTO is high; regarding disk faults, when the storage nodes generate disk faults and data are lost, copies are not automatically supplemented, and the storage nodes run in a state of degrading the number of copies for a long time, so that the risk of data loss is increased; Aiming at the problem that the operation efficiency of the graph database is low because the copy number of the traditional graph database is fixed and manual intervention is needed, no effective solution is proposed at present. Disclosure of Invention In order to solve the technical problems, the embodiment of the invention is expected to provide a method and a device for adaptively adjusting the copy based on the graph database, so as to at least solve the problem of low operation efficiency of the graph database caused by manual intervention required by fixed copy number of the traditional graph database. The technical scheme of the invention is realized as follows: The embodiment of the invention provides a method for adaptively adjusting copies based on a graph database, which comprises the steps of setting a corresponding expected copy value for each Zone in a cluster creation stage, detecting the copy state reported by a storage layer in real time to obtain PartView of current data copy distribution and StoreView of physical resource use conditions, comparing the copy number in PartView with the expected copy value, generating a copy change instance when the copy number is different from the expected copy value, selecting a target storage node through a filter and a speed limiter according to StoreView, executing the capacity expansion or contraction operation of the copies, and ending a copy adjustment process when the copy number is the same as the expected copy value. Optionally, the method further includes sending a metadata copy capacity reduction request to a raft group of the metadata service metad when the main area fails, performing member change operation on the raft group of the metadata service metad according to the metadata copy capacity reduction request to enable the metadata service metad of the spare area to become a leader node, sending a data copy capacity reduction request to the leader node after the metadata service metad of the spare area becomes the leader node, performing a capacity reduction operation on the leader node according to the data copy capacity reduction request so that the leader of all the data fragments is concentrated in the spare area, providing write service by the spare area, sending a metadata copy capacity expansion request to the metadata service metad after the main area is restored, for restoring the copy number of the metadata service metad, sending a data copy capacity expansion request to the metadata service metad of the spare area of the leader node after the copy number of the metadata service metad is restored, performing data copy capacity reduction operation on the leader node according to the data copy capacity request and a desired copy value, performing copy capacity reduction operation on the leader of all the data fragments so that the leader of the data fragments is concentrated in the spare area, and re-writing the desired copy value of the data fragments by the main area. Optionally, selecting the target storage node by the filter according to StoreView includes selecting the target storage node for carrying the new copy according to at least one factor filter, the at least one factor including isolation, node capacity and status, snapshot count, and copy filling efficiency. Optionally, maintaining each storage node through the speed limiter, wherein in the copy capacity expansion process, the allocation rate of selecting the storage node for the plurality of data fragments is controlled, so that the storage nodes are sequentially staggered and selected for uniform resource allocation when the plurality of data fragments are selected. Optionally, performing the copy expansion or contraction operation includes sending metadata copy adjustment instructions to the metadata service metad via ngctl tool, sending data copy adjustment instructions to the metadata service metad via ngql tool, and performing copy adjustment processing by the