CN-122027616-A - Self-adaptive file migration method and device based on service gateway and electronic equipment

CN122027616ACN 122027616 ACN122027616 ACN 122027616ACN-122027616-A

Abstract

The application relates to a self-adaptive file migration method based on a service gateway, which comprises the steps that the service gateway receives an access request for a target file and inquires a migration state of the target file in a file path mapping table, if the migration state is not migrated, the service gateway accesses a source storage service to obtain the target file, identifies one or more associated files with access correlation with the target file based on a preset association rule, adds unique identifiers of the target file and the associated files into a high-priority migration queue, executes a corresponding high-priority migration task, executes a low-priority migration task based on the self-adaptive migration rate, and migrates all files in the source storage service to the target storage service. According to the application, the service gateway is used for realizing the non-inductive migration of the heterogeneous storage service, and the load self-adaption and the associated prefetching strategy are utilized, so that the access delay is eliminated while the stability of the system is ensured, and the user experience is obviously improved.

Inventors

ZHU XINYI
WANG KAI
Lan Zhuhui

Assignees

前锦网络信息技术(上海)有限公司

Dates

Publication Date: 20260512
Application Date: 20251223

Claims (10)

1. The self-adaptive file migration method based on the service gateway is characterized by comprising the following steps of: The service gateway receives an access request aiming at a target file and inquires the migration state of the target file in a file path mapping table; If the migration state is not migrated, accessing a source storage service through a service gateway to acquire the target file, and identifying one or more associated files with access relevance to the target file based on a preset association rule; Adding unique identifiers of the target file and the associated file into a high-priority migration queue, and executing corresponding high-priority migration tasks; And executing a low-priority migration task based on the self-adaptive migration rate, and migrating the full files in the source storage service to the target storage service, wherein the self-adaptive migration rate is determined based on a system load value, and the system load value is used for quantifying the current resource occupation degree of the service gateway.
2. The adaptive file migration method of claim 1, wherein the adaptive migration rate determination based on a system load index comprises: Determining a system load value according to the current service request throughput and the disk IO delay index; if the system load value is greater than or equal to a preset load threshold value, suspending the execution of the low-priority migration task; And if the system load value is smaller than the load threshold value, determining the self-adaptive migration rate of the low-priority migration task according to the system load value and the load threshold value.
3. The adaptive file migration method of claim 2, wherein the adaptive migration rate is calculated using the following equation: Wherein, the In order to adapt the rate of migration, At the time of the highest rate of migration, As a result of the load threshold value in question, For the current value of the system load, The coefficients are adjusted for the rate.
4. The adaptive file migration method of claim 1, wherein the unique identifier is a hash feature value calculated based on file content, and wherein when performing high-priority and/or low-priority migration tasks, further comprising: before writing the target file into the target storage service, judging whether the target storage service has stock files with the same hash characteristic value or not; If so, directly establishing a mapping relation between the service path of the target file and the physical path of the stock file, and marking that migration is completed.
5. The adaptive file migration method of claim 4, further comprising: In response to establishing a mapping relationship between the business path of the target file and the physical path of the stock file, incrementing a counter field value; deleting the mapping relation between the business path of the target file and the physical path when receiving a deleting request for the target file, and subtracting one from the value of the counter field; When the value of the counter field returns to zero, the stock file is physically deleted from the target storage service.
6. The adaptive file migration method of claim 4, further comprising: determining a preset archiving time based on a lifecycle policy of the target file; generating a management soft path pointing to the physical path of the stock file in the time aggregation directory identified by the preset filing time; and responding to the system time reaching the preset archiving time, and executing archiving processing on the corresponding physical files based on the management soft path.
7. The adaptive file migration method according to claim 1, wherein the preset association rule includes: identifying a file under the same directory as the target file in the source storage service as an associated file, and/or Identifying a file having an upload timestamp difference with the target file within a preset time window as an associated file, and/or And identifying the file with the file ID close to the file ID of the target file as an associated file.
8. An adaptive file migration apparatus based on a service gateway, comprising: the service gateway is used for receiving an access request aiming at a target file and inquiring the migration state of the target file in the file path mapping table; The identification module is used for accessing a source storage service through a service gateway to acquire the target file if the migration state is not migrated, and identifying one or more associated files with access correlation with the target file based on a preset association rule; the first migration module is used for adding the unique identifiers of the target file and the associated file into a high-priority migration queue and executing corresponding high-priority migration tasks; and the second migration module is used for executing a low-priority migration task based on the self-adaptive migration rate, and migrating the full files in the source storage service to the target storage service, wherein the self-adaptive migration rate is determined based on a system load value, and the system load value is used for quantifying the current resource occupation degree of the service gateway.
9. An electronic device comprising a processor and a memory, the memory having stored thereon a set of computer program instructions which, when executed by the processor, implement the service gateway-based adaptive file migration method of any one of claims 1-7.
10. A computer program product comprising a set of computer program instructions which, when executed by a processor, implement the service gateway based adaptive file migration method of any one of claims 1 to 7.

Description

Self-adaptive file migration method and device based on service gateway and electronic equipment Technical Field The present invention relates to the field of computer technologies, and in particular, to a service gateway-based adaptive file migration method, apparatus, electronic device, and computer program product. Background With the rapid development of internet traffic, the data volume of enterprises has increased exponentially. To meet business expansion, performance optimization, or cost control requirements, enterprises often need to migrate large amounts of file data from old storage systems to new storage systems. However, it is difficult to combine the service transparency and the system stability in the data migration process in the current technology. On one hand, the history file path is often solidified in a service layer, but the traditional migration tool only focuses on data carrying but can not solve the mapping problem after the storage address is changed, so that the bottom storage change forces the upper service to carry out code reconstruction, and smooth transition of service noninductivity is difficult to realize, on the other hand, large-scale data migration is taken as a high-resource consumption task, and under the condition of lacking a fine scheduling mechanism, resource contention is easily generated with a real-time service request, so that system response delay and even service unavailability are caused, and normal operation of the system is influenced. Disclosure of Invention In view of the above, embodiments of the present application provide a service gateway-based adaptive file migration method, apparatus, electronic device, and computer program product, which are used for solving at least one technical problem. The embodiment of the application provides a self-adaptive file migration method based on a service gateway, which comprises the steps that the service gateway receives an access request for a target file and inquires a migration state of the target file in a file path mapping table, if the migration state is not migrated, a source storage service is accessed through the service gateway to obtain the target file, one or more associated files with access correlation with the target file are identified based on a preset association rule, unique identifiers of the target file and the associated files are added into a high-priority migration queue, corresponding high-priority migration tasks are executed, a low-priority migration task is executed based on self-adaptive migration rate, the whole files in the source storage service are migrated to the target storage service, the self-adaptive migration rate is determined based on a system load value, and the system load value is used for quantifying the current resource occupation degree of the service gateway. The self-adaptive file migration method comprises the steps of determining a system load value according to current service request throughput and a disk IO delay index, suspending execution of a low-priority migration task if the system load value is greater than or equal to a preset load threshold, and determining the self-adaptive migration rate of the low-priority migration task according to the system load value and the load threshold if the system load value is less than the load threshold. The adaptive file migration method as described above calculates the adaptive migration rate using the following equation: Wherein, the In order to adapt the rate of migration,At the time of the highest rate of migration,As a result of the load threshold value in question,For the current value of the system load,The coefficients are adjusted for the rate. The self-adaptive file migration method comprises the steps that the unique identifier is a hash characteristic value calculated based on file content, when a high-priority and/or low-priority migration task is executed, whether a stock file with the same hash characteristic value exists in a target storage service or not is judged before the target file is written into the target storage service, if so, a mapping relation between a service path of the target file and a physical path of the stock file is directly established, and migration completion is marked. The self-adaptive file migration method further comprises the steps of adding one to a value of a counter field in response to establishing a mapping relation between a service path of the target file and a physical path of the stock file, deleting the mapping relation between the service path of the target file and the physical path when a deleting request for the target file is received, subtracting one to the value of the counter field, and physically deleting the stock file from the target storage service when the value of the counter field is zero. The self-adaptive file migration method further comprises the steps of determining preset filing time based on the life cycle strategy of the target file, gene