CN-122027624-A - Data migration method, device, equipment and storage medium across storage equipment

CN122027624ACN 122027624 ACN122027624 ACN 122027624ACN-122027624-A

Abstract

The invention discloses a data migration method across storage equipment, a device, equipment and a storage medium, which are applied to metadata servers and relate to the technical field of cloud computing, and the method comprises the steps of locking metadata of data fragments to be migrated, and sending migration preparation instructions to a first data server and a second data server, so that the first data server asynchronously pushes the data fragments to be migrated to the second data server based on the migration preparation instructions; and when the data fragments to be migrated are migrated, determining a target access server from the first data server and the second data server based on a data access request of the client for the data fragments to be migrated, so that the client can access the data fragments to be migrated based on the target access server, notifying the first data server to delete the data copies of the data fragments to be migrated, and updating the data positions of the data fragments to be migrated into the second data server. In this way, seamless migration of the distributed storage system may be supported.

Inventors

YANG YONG
SU ZHIHENG

Assignees

济南浪潮数据技术有限公司

Dates

Publication Date: 20260512
Application Date: 20260330

Claims (10)

1. A method for migrating data across storage devices, applied to a metadata server, comprising: The method comprises the steps of locking metadata of a data fragment to be migrated, and sending a migration preparation instruction to a first data server and a second data server, so that the first data server asynchronously pushes the data fragment to be migrated to the second data server based on the migration preparation instruction; When the data fragments to be migrated are migrated, determining a target access server from the first data server and the second data server based on a data access request of a client for the data fragments to be migrated, so that the client performs data access on the data fragments to be migrated based on the target access server; and notifying the first data server to delete the data copy of the data fragment to be migrated, and updating the data position of the data fragment to be migrated to the second data server.
2. The method for migrating data across storage devices according to claim 1, wherein said locking metadata of the data fragments to be migrated and sending migration preparation instructions to the first data server and the second data server comprises: acquiring a data migration task, and locking metadata of data fragments to be migrated based on the data migration task; And setting the fragmentation state of the data fragments to be migrated from a normal state to a preparation state, and sending migration preparation instructions to the first data server and the second data server so that the second data server allocates the storage space of the data fragments to be migrated.
3. The method of data migration across storage devices of claim 1, wherein the first data server asynchronously pushing the data fragments to be migrated to the second data server based on the migration preparation instructions, comprising: setting the fragmentation state of the data fragments to be migrated from a preparation state to a migration state; And starting asynchronous pushing of the data fragments to be migrated to the second data server by using the first data server and based on the migration preparation instruction.
4. The method for migrating data across storage devices according to claim 3, wherein after said asynchronously pushing said data fragments to be migrated to said second data server using said first data server and based on said migration preparation instruction, further comprising: acquiring a data writing request of a client for the data fragments to be migrated, and acquiring the states of the data fragments to be migrated; If the data fragments to be migrated are in a migration state, writing the data fragments to be migrated on the first data server based on the data writing request, and recording operation content corresponding to the writing operation into a preset migration log so as to reproduce the writing operation based on the preset migration log; And if the data to be migrated is fragmented into a data redirection state, performing writing operation on the data to be migrated on the first data server and the second data server based on the data writing request and by utilizing a data double-writing mechanism.
5. The method for migrating data across storage devices according to claim 1, wherein when the migration of the data shard to be migrated is completed, determining a target access server from the first data server and the second data server based on a data access request of a client for the data shard to be migrated, so that the client performs data access on the data shard to be migrated based on the target access server, includes: when the data to be migrated is segmented, setting the segmentation state of the data to be migrated from the migration state to the data redirection state; acquiring a data access request of a client for the data fragments to be migrated; if the data access request is a data write request, simultaneously writing data to be written corresponding to the data write request into the first data server and the second data server by utilizing the data to be written corresponding to the data write request so as to complete the data write request; If the data access request is a data read request, simultaneously sending server addresses corresponding to the first data server and the second data server to the client; The client reads the data fragments of the data to be migrated based on the server address corresponding to the second data server; and if the client data reading fails, reading the data of the data fragments to be migrated based on the server address corresponding to the first data server.
6. The method for migrating data across storage devices according to claim 1, wherein before notifying the first data server to delete the data copy of the data shard to be migrated and updating the data location of the data shard to be migrated to the second data server, further comprises: judging whether the data in the first data server and the second data server are consistent; And if the data in the first data server and the data in the second data server are consistent, setting the fragmentation state of the data fragments to be migrated from a data redirection state to a cleaning state.
7. The method of transferring data across storage devices according to any one of claims 1 to 6, wherein the notifying the first data server to delete the data copy of the data shard to be transferred and update the data location of the data shard to be transferred to the second data server includes: notifying the first data server to delete the data copy of the data fragment to be migrated; Setting the data position of the data fragment to be migrated as the second data server, and unlocking the metadata of the data fragment to be migrated; And setting the fragmentation state of the data fragments to be migrated to be a normal state.
8. A data migration apparatus across storage devices, characterized by being applied to a metadata server, comprising: The system comprises an instruction generation module, a first data server, a second data server, a data processing module and a data processing module, wherein the instruction generation module is used for locking metadata of a data fragment to be migrated and sending a migration preparation instruction to the first data server and the second data server so that the first data server asynchronously pushes the data fragment to be migrated to the second data server based on the migration preparation instruction; the data migration module is used for determining a target access server from the first data server and the second data server based on a data access request of a client for the data fragments to be migrated when the data fragments to be migrated are migrated, so that the client can access the data fragments to be migrated based on the target access server; and the address matching module is used for notifying the first data server to delete the data copy of the data fragment to be migrated and updating the data position of the data fragment to be migrated to the second data server.
9. An electronic device, comprising: A memory for storing a computer program; a processor for executing the computer program to perform the steps of the data migration method across storage devices according to any one of claims 1 to 7.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the data migration method across storage devices according to any of claims 1 to 7.

Description

Data migration method, device, equipment and storage medium across storage equipment Technical Field The present invention relates to the field of cloud computing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for migrating data across storage devices. Background In modern data centers, distributed storage systems carry the data storage requirements of core services. The global file system serves as an upper layer abstraction, shields the complexity of the bottom layer physical equipment and provides a single and unified file view for a user. Typical global file systems include Ceph FS (CEPH FILE SYSTEM, the Ceph file system), lustre (a large-scale cluster-oriented parallel file system), HDFS (Hadoop Distributed FILE SYSTEM, the Hadoop distributed file system), and the like. However, the operation of the system is not a constant one, and the following scenario frequently drives the requirement of data migration, namely hardware upgrade and capacity expansion, namely, by replacing new and old storage devices or adding new storage nodes in a cluster, partial data needs to be migrated to the new devices to fully utilize the performance or capacity of the new devices, load balancing, when some storage nodes are excessively loaded due to hot spot data, I/O pressure and storage space utilization rate of the whole cluster need to be balanced by migrating the data, storage layering, and migration is performed between storage media with different performances, such as SSDs (Solid-state disks) STATE DRIVE and HDDs (HARD DISK DRIVE, namely, mechanical hard disks) according to the access frequency (hot, warm and cold) of the data, so as to optimize the total possession cost TCO (Total Cost of Ownership, namely, the total possession cost), fault prevention and isolation, and early migration of the data on some devices to other healthy nodes when possible faults of the devices are predicted or the devices need to be maintained in a line. The traditional data migration method, such as updating metadata after direct copying, has obvious limitations that data involved in the migration period may not be available, and large-scale data migration occupies a large amount of network and disk I/O resources, so that the foreground service performance is seriously affected. Therefore, how to ensure the strong consistency of data and the performance of foreground service in the migration process is a problem to be solved by those skilled in the art. Disclosure of Invention The embodiment of the invention aims to provide a data cross-storage device migration method, a device, equipment and a storage medium, which can support seamless migration of a distributed storage system and ensure smooth service operation. The specific scheme is as follows: in a first aspect, the present application discloses a data migration method across storage devices, which is applied to a metadata server, and includes: The method comprises the steps of locking metadata of data fragments to be migrated, and sending migration preparation instructions to a first data server and a second data server, so that the first data server asynchronously pushes the data fragments to be migrated to the second data server based on the migration preparation instructions; when the data fragments to be migrated are migrated, determining a target access server from the first data server and the second data server based on a data access request of the client for the data fragments to be migrated, so that the client performs data access on the data fragments to be migrated based on the target access server; And notifying the first data server to delete the data copy of the data fragment to be migrated, and updating the data position of the data fragment to be migrated to the second data server. Optionally, locking metadata of the data fragment to be migrated, and sending a migration preparation instruction to the first data server and the second data server, including: acquiring a data migration task, and locking metadata of the data fragments to be migrated based on the data migration task; And setting the fragmentation state of the data fragments to be migrated from the normal state to the preparation state, and sending migration preparation instructions to the first data server and the second data server so that the second data server allocates the storage space of the data fragments to be migrated. Optionally, the first data server asynchronously pushes the data fragments to be migrated to the second data server based on the migration preparation instruction, including: setting the fragmentation state of the data fragments to be migrated from the preparation state to the migration state; and starting asynchronous pushing of the data fragments to be migrated to the second data server by utilizing the first data server and based on the migration preparation instruction. Optionally, after starting asynchronous pushing the data fragme