CN-115934640-B - Data storage method, system, electronic equipment and storage medium

CN115934640BCN 115934640 BCN115934640 BCN 115934640BCN-115934640-B

Abstract

The invention provides a data storage method, a system, electronic equipment and a storage medium, wherein when a client receives a ticket fed back by an authorization server, connection is established with an HDFS system, a remote process access request is sent to the HDFS system, when the HDFS system determines that a file to be migrated with a matched name of the file to be migrated in the remote access request exists, at least one data node information corresponding to the file to be migrated is acquired, a target link with the shortest distance between a target data node corresponding to each data node information and the client is determined, the client communicates with the corresponding target data node according to the corresponding target link, acquires a data block corresponding to the file to be migrated from the target data node, and transmits the data block to a blue light storage system connected with the HDFS system until all the data blocks related to the file to be migrated are transmitted to the blue light storage system. The invention can solve the problems that the operation risk is easy to occur and the corresponding operation cost is increased in the data migration process.

Inventors

Lv Yini
YANG YUAN
CAO MINGXUAN
ZHAO WENTAO
LI WEIWEI
Lin Enai
LOU FENG
MA XIAOPING
HAN CHAO
XI WEIHONG
GAO YUN
DENG GANG
MEN YUSEN
LI RUI
YANG JUN
LIANG CE

Assignees

中国民航信息网络股份有限公司

Dates

Publication Date: 20260505
Application Date: 20221229

Claims (8)

1. A data storage method, applied to a client, the method comprising: When a ticket fed back by an authorization server is received, connection is established with an HDFS system, and a remote process access request is sent to the HDFS system, so that when the HDFS system determines that a file to be migrated with a matched name of the file to be migrated in the remote process access request exists, at least one data node information corresponding to the file to be migrated is obtained, and a target link with the shortest distance between a target data node corresponding to each data node information and the client is determined; According to the corresponding target link, the data blocks corresponding to the files to be migrated are obtained from the target data nodes, and the data blocks are transmitted to a blue light storage system connected with the HDFS system until all the data blocks related to the files to be migrated are transmitted to the blue light storage system; After all the data blocks related to the file to be migrated are transmitted to the blue-ray storage system, the method further includes: Acquiring first data total amount information of the file to be migrated from the HDFS system, and acquiring second data total amount information of the file to be migrated in the blue-ray storage system; Comparing the first data total amount information with the second data total amount information; If the first data total amount information is consistent with the second data total amount information, determining that the file to be migrated is complete in migration; If the first data total amount information and the second data total amount information are inconsistent, determining that the migration of the file to be migrated is incomplete, and outputting corresponding error reporting information; the generation process of the verification information comprises the steps of sending the authorization ticket request to the authentication server, enabling the authentication server to obtain a corresponding authorization ticket according to a user name, encrypting the authorization ticket by utilizing a password, and sending the obtained encrypted information to the client, wherein the authorization ticket request comprises the user name and the password, decrypting the encrypted information by utilizing the password, obtaining the authorization ticket, and generating the verification information according to TICKET GRANTING TICKET.
2. The method of claim 1, wherein said transmitting said data block to a blue light storage system coupled to said HDFS system comprises: Creating a storage bucket corresponding to the file to be migrated on the blue light storage system, and acquiring a target URL corresponding to the storage bucket; and transmitting the data block to a storage bucket on the blue light storage system according to the target URL.
3. The method according to claim 1, wherein the method further comprises: After the data block is transmitted to a blue light storage system connected with the HDFS system, calculating a check value of the data block, and judging whether the check value is consistent with a check value of a check bit in the data block or not; if the check value is consistent with the check value of the check bit in the data block, determining that the data block transmission is completed; If the check value is inconsistent with the check value of the check bit in the data block, determining that the data block fails to be transmitted, and outputting corresponding error reporting information.
4. The method according to claim 1, wherein the method further comprises: when a reverse migration request sent by a user is received, corresponding reverse migration information is obtained according to the reverse migration request, wherein the reverse migration information comprises an object storage URL, an access key and an access key; judging whether a corresponding reverse migration file exists in the blue-ray storage system according to the reverse migration information; And if the reverse migration file exists in the blue light storage system, migrating the reverse migration file on the blue light storage system to the HDFS system.
5. The method according to claim 4, wherein the method further comprises: According to the access key and the access key, carrying out identity authentication on the user; if the identity verification of the user is passed, corresponding user file directory read-write permission information is obtained; wherein migrating the reverse migration file on the blue light storage system to the HDFS system includes: And if the read-write permission information of the user file directory is permission for reading and writing, migrating the reverse migration file on the blue-ray storage system to the HDFS system.
6. A data storage system for application to a client, the system comprising: The remote process access request sending unit is used for establishing connection with an HDFS system when receiving a ticket fed back by an authorization server, and sending a remote process access request to the HDFS system, so that when the HDFS system determines that a file to be migrated with a matched name of the file to be migrated exists in the remote process access request, at least one data node information corresponding to the file to be migrated is obtained, and a target link with the shortest distance between a target data node corresponding to each data node information and the client is determined; The transmission unit is used for communicating with the corresponding target data node according to the corresponding target link, acquiring a data block corresponding to the file to be migrated from the target data node, and transmitting the data block to a blue light storage system connected with the HDFS system until all data blocks related to the file to be migrated are transmitted to the blue light storage system; after executing the transmission unit, the data storage system further includes: the data total amount information acquisition unit is used for acquiring first data total amount information of the file to be migrated from the HDFS system and acquiring second data total amount information of the file to be migrated from the blue-ray storage system; the comparison unit is used for comparing the first data total amount information with the second data total amount information; the second determining unit is used for determining that the file to be migrated is complete if the first data total amount information is consistent with the second data total amount information; The second output unit is used for determining that the file to be migrated is incomplete in migration if the first data total amount information is inconsistent with the second data total amount information, and outputting corresponding error reporting information; The verification information generating unit comprises an authorization ticket request transmitting unit and a verification information generating subunit; The authorization ticket request sending unit is used for sending the authorization ticket request to the authentication server so that the authentication server obtains a corresponding authorization ticket according to a user name, encrypts the authorization ticket by utilizing a password and sends the obtained encrypted information to the client; wherein the authorization ticket request includes the user name and a password; the verification information generation subunit is configured to decrypt the encrypted information by using the password to obtain the authorization ticket, and generate the verification information according to TICKET GRANTING TICKET.
7. An electronic device comprising a processor and a memory, the processor and the memory being connected by a communication bus, wherein the processor is configured to invoke and execute a program stored in the memory, and wherein the memory is configured to store a program configured to implement the data storage method according to any one of claims 1-5.
8. A computer-readable storage medium comprising, the computer-readable storage medium has stored therein computer-executable instructions, computer executable instructions for performing the data storage method of any of claims 1-5.

Description

Data storage method, system, electronic equipment and storage medium Technical Field The present invention relates to the field of computer technologies, and in particular, to a data storage method, a system, an electronic device, and a storage medium. Background As society evolves and time accumulates, a vast amount of information is generated, which is typically stored digitally in a corresponding device. Because of different application scenes of different data, the time, inquiry, update speed and other related requirements of the different data are also greatly different, so that the data which need to be frequently inquired or modified can be stored in a OLTP (On Line Transaction Processing) database system which can provide rapid inquiry and update operation, and the data which are not modified for statistical analysis are stored in an OLAP (On LINE ANALYTICAL Processing) data warehouse and a large data system. However, most of the bottom storage media of the current data warehouse and big data system are solid state hard disk, mechanical hard disk, etc., the service life is generally low, and the data warehouse and big data system are not suitable for long-term data storage, and the stored data needs to be migrated to a new storage device when the validity period is approaching each time, so that operation risks are easy to occur in the data migration process, and the corresponding operation cost is increased. Disclosure of Invention In view of the above, the present invention provides a data storage method, system, electronic device and storage medium, so as to solve the problems that the existing storage device needs to migrate stored data to a new storage device each time approaching to the validity period due to the limited storage period, resulting in easy occurrence of operation risk in the process of data migration and increase of corresponding operation cost. The first aspect of the invention discloses a data storage method, which is applied to a client, and comprises the following steps: When a ticket fed back by an authorization server is received, connection is established with an HDFS system, and a remote process access request is sent to the HDFS system, so that when the HDFS system determines that a file to be migrated with a matched name of the file to be migrated in the remote access request exists, at least one data node information corresponding to the file to be migrated is obtained, and a target link with the shortest distance between a target data node corresponding to each data node information and the client is determined; And communicating with the corresponding target data node according to the corresponding target link, acquiring a data block corresponding to the file to be migrated from the target data node, and transmitting the data block to a blue light storage system connected with the HDFS system until all data blocks related to the file to be migrated are transmitted to the blue light storage system, wherein the file to be migrated is data which does not need modification, has non-real time query requirement and has long storage time requirement. A second aspect of the present invention discloses a data storage system, the data storage system being applied to a client, the system comprising: The system comprises a remote process access request sending unit, an authentication server, a client and a verification information generating unit, wherein the remote process access request sending unit is used for establishing connection with the HDFS system when receiving a ticket fed back by the authentication server and sending a remote process access request to the HDFS system, so that when the HDFS system determines that a file to be migrated with a matched name of the file to be migrated in the remote access request exists, at least one data node information corresponding to the file to be migrated is obtained, and a target link with the shortest distance between a target data node corresponding to each data node information and the client is determined; The transmission unit is used for communicating with the corresponding target data node according to the corresponding target link, acquiring a data block corresponding to the file to be migrated from the target data node, and transmitting the data block to a blue light storage system connected with the HDFS system until all data blocks related to the file to be migrated are transmitted to the blue light storage system, wherein the file to be migrated is data which does not need modification, has non-real time query requirement and has long storage time requirement. The third aspect of the invention provides an electronic device comprising a processor and a memory, wherein the processor and the memory are connected through a communication bus, the processor is used for calling and executing a program stored in the memory, and the memory is used for storing the program, and the program is used for realizing the data storage