CN-115664826-B - Data file encryption method, device, computing equipment and storage medium

CN115664826BCN 115664826 BCN115664826 BCN 115664826BCN-115664826-B

Abstract

The application discloses a data file encryption method, a device, a computing device and a storage medium, wherein the method comprises the steps of receiving a file uploading request of a data file, and respectively storing a plurality of data fragments of the data file into a plurality of data nodes in a distributed storage system; the method comprises the steps of obtaining load information of a data node stored in any data fragment of a data file, determining an object for executing encryption according to the load information of the data node stored in the data fragment, and controlling the data node to execute encryption on the data fragment if the object for executing encryption is determined to be the data node stored in the data fragment. By the method, only a part of computing resources are reserved dynamically by the data node, and under the condition of low load of the data node, encryption processing of the data fragments is completed by the data node, and the data fragments are not required to be loaded to an external computing node for encryption processing, so that network bandwidth resources can be saved, and the condition of overlarge throughput of the data node can be avoided.

Inventors

LIANG SONGTAO
LUO XIAOFAN
MIAO CHUNJIE

Assignees

上海哔哩哔哩科技有限公司

Dates

Publication Date: 20260508
Application Date: 20221028

Claims (15)

1.A method of encrypting a data file, comprising: Receiving a file uploading request of a data file, and respectively storing a plurality of data fragments of the data file into a plurality of data nodes in a distributed storage system; load information of a data node stored in any data fragment of the data file is obtained; Determining an object for executing encryption processing according to the load information of the data node stored by the data fragment; And if the data fragment is any other data fragment except the 1 st data fragment, splicing the tail data corresponding to the previous data fragment with the data of the data fragment, and carrying out encryption processing on encryption basic data comprising the spliced data and the hash intermediate result corresponding to the previous data fragment to obtain the encryption processing result of the data fragment, wherein the encryption processing result of the data fragment comprises the hash intermediate result obtained by carrying out encryption processing on a plurality of packets or the hash intermediate result and the tail data obtained by carrying out encryption processing on a plurality of packets, and the plurality of packets and the tail data are obtained by carrying out grouping on the encryption basic data.
2. The method of claim 1, wherein after the determining the object to perform the encryption process, the method further comprises: and if the object for executing the encryption processing is determined to be a computing node, controlling the computing node to read the data fragment from the data node and load the data fragment to the local computing node so as to execute the encryption processing on the data fragment.
3. The method of claim 1, wherein prior to receiving the file upload request for the data file, the method further comprises: Receiving a plurality of data fragments of a data file uploaded in a breakpoint continuous transmission mode; The plurality of data fragments are generated by dividing the data file according to a first preset size.
4. The method of claim 1, wherein the load information comprises CPU load information and/or disk load information; the determining the object for executing the encryption processing according to the load information of the data node stored in the data fragment further comprises: determining that the object for executing encryption processing is the data node under the condition that the CPU load information of the data node stored by the data fragment is lower than a first preset value and/or the disk load information is lower than a second preset value; and determining the object for executing encryption processing as a computing node under the condition that the CPU load information of the data node stored by the data fragment is higher than a first preset value and/or the disk load information is higher than a second preset value.
5. The method of any of claims 1-4, wherein the controlling the data node to encrypt the data slice further comprises: And creating a first computing task, and sending the first computing task to the data node so as to enable the data node to run the first computing task, wherein the first computing task is used for carrying out encryption processing on the data fragments.
6. The method of claim 2, wherein the controlling the computing node to read the data fragment from the data node and load it locally to the computing node to encrypt the data fragment further comprises: and creating a second computing task, and sending the second computing task to a computing node so that the computing node can run the second computing task, wherein the second computing task is used for reading the data fragments from the data node, loading the data fragments to the local computing node and carrying out encryption processing on the data fragments.
7. The method of any of claims 1-4, wherein said encrypting the data slice further comprises: Performing encryption processing according to the data of the data fragment and the encryption processing result of the previous data fragment of the data fragment to obtain the encryption processing result of the data fragment; and if the data fragment is the 1 st data fragment in the data file, the encryption processing result of the previous data fragment of the data fragment is a preset encryption processing result.
8. The method of claim 7, wherein if the data fragment is the 1 st data fragment in the data file, performing encryption processing according to the data of the data fragment and the encryption processing result of the previous data fragment of the data fragment, and obtaining the encryption processing result of the data fragment further comprises: If the size of the data fragments is an integer multiple of a second preset size, grouping the data fragments according to the second preset size to obtain a plurality of groups; if the size of the data fragments is not an integer multiple of the second preset size, grouping the data fragments according to the second preset size to obtain a plurality of groups and tail data; and performing encryption operation according to the plurality of packets and the preset encryption processing result to obtain a hash intermediate result.
9. The method of claim 7, wherein if the data slice is any other data slice except the 1 st data slice, the encrypting according to the data of the data slice and the encryption result of the previous data slice further comprises: And under the condition that the encryption processing result of the previous data fragment comprises a hash intermediate result but does not comprise tail data, carrying out encryption processing according to the data of the data fragment and the hash intermediate result corresponding to the previous data fragment to obtain the encryption processing result of the data fragment.
10. The method of claim 9, wherein the method further comprises: And extracting a hash intermediate result contained in the encryption processing result of the last data fragment of the data file, and storing the extracted hash intermediate result as the encryption processing result of the data file.
11. The method of claim 2, wherein after determining that the object performing the encryption process is a compute node, the method further comprises: Screening target computing nodes from the computing node cluster according to the load information of each computing node in the computing node cluster; The controlling the computing node to read the data fragment from the data node and load the data fragment locally to the computing node to encrypt the data fragment further comprises: And controlling the target computing node to read the data fragments from the data node and load the data fragments to the local computing node so as to encrypt the data fragments.
12. A data file encryption apparatus comprising: the acquisition module is suitable for receiving a file uploading request of a data file and storing a plurality of data fragments of the data file into a plurality of data nodes in the distributed storage system respectively; the decision module is suitable for acquiring the load information of the data node stored in any data fragment of the data file, and determining an object for executing encryption processing according to the load information of the data node stored in the data fragment; The processing module is suitable for controlling the data node to encrypt the data fragments if the object for executing encryption processing is determined to be the data node stored by the data fragments, wherein if the data fragments are any other data fragments except the 1 st data fragment, the processing module is used for splicing the tail data corresponding to the previous data fragment and the data of the data fragments when the encryption processing result of the previous data fragment comprises a hash intermediate result and the tail data, and encrypting the encryption basic data comprising the spliced data and the hash intermediate result corresponding to the previous data fragment to obtain the encryption processing result of the data fragments, and the encryption processing result of the data fragments comprises the hash intermediate result obtained by encrypting a plurality of packets or the hash intermediate result and the tail data obtained by encrypting a plurality of packets, and the plurality of packets and the tail data are obtained by grouping the encryption basic data.
13. A computing device comprising a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface completing communication with each other over the communication bus; The memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform operations corresponding to the data file encryption method according to any one of claims 1 to 11.
14. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the data file encryption method of any one of claims 1-11.
15. A computer program product comprising at least one executable instruction for causing a processor to perform operations corresponding to the data file encryption method of any one of claims 1 to 11.

Description

Data file encryption method, device, computing equipment and storage medium Technical Field The present application relates to the field of data processing technologies, and in particular, to a data file encryption method, device, computing equipment, and storage medium. Background In order to ensure that the data is not corrupted or tampered with after the file is uploaded to the distributed storage system, it is often necessary to encrypt the file via an encryption algorithm. At present, the data encryption is generally carried out in a mode of memory computation separation, namely a memory node and a computing node are mutually independent, the memory node is dedicated to storing data, and the computing node is dedicated to encryption computation, so that the data is firstly required to be pulled from a memory system and then sent to the computing node through a gateway of the memory system, the pulled data is encrypted by the computing node, and the memory computation separation mode has the advantage of service decoupling. However, the method of computational separation has many defects, including low encryption processing efficiency, high consumption of network bandwidth resources, increased throughput of data nodes, and the like. Disclosure of Invention The embodiment of the application aims to provide a data file encryption method, a device, a computing device and a storage medium, so as to solve the problems of low encryption processing efficiency, high network bandwidth consumption, increased throughput of data nodes and the like caused by the storage and calculation separation mode. According to an aspect of the present application, there is provided a data file encryption method including: receiving a file uploading request of a data file, and respectively storing a plurality of data fragments of the data file into a plurality of data nodes in a distributed storage system; load information of a data node stored in any data fragment of the data file is obtained; Determining an object for executing encryption processing according to the load information of the data node stored by the data fragment; And if the object for executing the encryption processing is determined to be the data node stored in the data fragment, controlling the data node to execute the encryption processing on the data fragment. Optionally, after determining the object to perform the encryption process, the method further comprises: and if the object for executing the encryption processing is determined to be the computing node, the computing node is controlled to read the data fragment from the data node and load the data fragment to the local computing node so as to execute the encryption processing on the data fragment. Optionally, before receiving the file upload request of the data file, the method further comprises: Receiving a plurality of data fragments of a data file uploaded in a breakpoint continuous transmission mode; the plurality of data fragments are generated by dividing the data file according to a first preset size. Optionally, the load information includes CPU load information and/or disk load information; Determining an object for executing encryption processing according to the load information of the data node stored in the data fragment further comprises: determining that the object for executing encryption processing is the data node under the condition that the CPU load information of the data node stored by the data fragment is lower than a first preset value and/or the disk load information is lower than a second preset value; and determining the object for executing encryption processing as a computing node under the condition that the CPU load information of the data node stored by the data fragment is higher than a first preset value and/or the disk load information is higher than a second preset value. Optionally, controlling the data node to encrypt the data slice further includes: and creating a first computing task, and sending the first computing task to the data node so that the data node can run the first computing task, wherein the first computing task is used for encrypting the data fragments. Optionally, controlling the computing node to read the data fragment from the data node and load the data fragment locally to the computing node to encrypt the data fragment further comprises: And creating a second computing task, and sending the second computing task to the computing node for the computing node to run the second computing task, wherein the second computing task is used for reading the data fragments from the data node, loading the data fragments to the local computing node and carrying out encryption processing on the data fragments. Optionally, encrypting the data slice further includes: Performing encryption processing according to the data of the data fragment and the encryption processing result of the previous data fragment of the data fragment to obtain the encryption proces