Search

CN-116414771-B - Data migration method, device, electronic equipment and storage medium

CN116414771BCN 116414771 BCN116414771 BCN 116414771BCN-116414771-B

Abstract

The embodiment discloses a data migration method, a device, equipment, a system and a computer storage medium, wherein the method comprises the steps of receiving a character separation value CSV file read by other processing units of the Nifi cluster; the method comprises the steps of mapping text content of a CSV file into Record records, preprocessing the Record records to obtain preprocessed Record records, and storing the preprocessed Record records into a database.

Inventors

  • ZHANG KAIWEN
  • FENG JUNLAN

Assignees

  • 中国移动通信有限公司研究院
  • 中国移动通信集团有限公司

Dates

Publication Date
20260512
Application Date
20211230

Claims (6)

  1. 1. A data migration method, applied to a service processing unit in a Nifi cluster, the method comprising: Receiving a character separation value CSV file read by other processing units of the Nifi clusters; Mapping text content of the CSV file into Record records according to Avro format fields defined in Record readers of the Nifi clusters; adding a fixed prefix to a variable obtained by field splicing in the Record so as to identify the variable; And processing the identified variables according to the expression language grammar of the Nifi clusters to obtain preprocessed Record, and storing the preprocessed Record into a database.
  2. 2. The data migration method of claim 1, wherein the processing the identified variables according to the Nifi clusters' expression language grammar to obtain the preprocessed Record includes: And extracting and converting the variables according to the expression language grammar to obtain the preprocessed Record.
  3. 3. The data migration method of claim 2, wherein said performing variable extraction and data conversion on said identified variables according to said expression language grammar to obtain said preprocessed Record comprises: Identifying the identified variable as a variable using the expression language grammar in a onScheduled method; And when the onTrigger method is called, extracting and converting the variables through the expression language grammar, and obtaining the preprocessed Record.
  4. 4. A data migration device is characterized by being applied to service processing units in Nifi clusters, and comprises a receiving module, a first processing module and a second processing module, The receiving module is used for receiving the character separation value CSV file read by other processing units of the Nifi clusters; A first processing module, configured to map text content of the CSV file to Record according to Avro format fields defined in Record readers of the Nifi clusters; The second processing module is used for adding a fixed prefix to the variable obtained by field splicing in the Record so as to identify the variable; And processing the identified variables according to the expression language grammar of the Nifi clusters to obtain preprocessed Record, and storing the preprocessed Record into a database.
  5. 5. An electronic device comprising a processor and a memory for storing a computer program capable of running on the processor, wherein, The processor is configured to run the computer program to perform the data migration method of any one of claims 1 to 3.
  6. 6. A computer storage medium having stored thereon a computer program, which when executed by a processor implements the data migration method of any of claims 1 to 3.

Description

Data migration method, device, electronic equipment and storage medium Technical Field The present application relates to the field of data migration technologies, and in particular, to a data migration method, a data migration device, an electronic device, and a storage medium. Background In the related art, 7 processing units (processors) are needed for implementing data migration of a character segmentation value (Comma-SEPARATED VALUES, CSV) file in a Nifi cluster, and the processing units are involved in reading, filtering, mapping text and database fields, preprocessing and the like of the file, each processing unit independently runs in a Nifi cluster, and in the migration process of the CSV file, each processing unit needs to consume a large amount of disk read-write and network transmission resources, so that the data migration efficiency of the CSV file is reduced. Disclosure of Invention The embodiment of the application provides a data migration method, which can reduce disk read-write and network resource consumption caused when data migration is carried out on CSV files, and improves the efficiency of data migration. The embodiment of the application provides a data migration method, which comprises the following steps: Receiving CSV files read by other processing units of the Nifi clusters; mapping the text content of the CSV file into Record records; Preprocessing the Record to obtain a preprocessed Record, and storing the preprocessed Record into a database. In some embodiments, the preprocessing the Record to obtain a preprocessed Record includes: identifying variables obtained by field splicing in the Record; and processing the identified variables according to the expression language grammar of the Nifi clusters to obtain the preprocessed Record. In some embodiments, the identifying the variable obtained by field concatenation in the Record includes: and adding a fixed prefix to the variable to realize identification of the variable. In some embodiments, the processing the identified variable according to the Nifi clusters' expression language grammar to obtain the preprocessed Record includes: And extracting and converting the variables according to the expression language grammar to obtain the preprocessed Record. In some embodiments, the extracting and converting the variables according to the expression language grammar to obtain the preprocessed Record includes: Identifying the identified variable as a variable using the expression language grammar in a onScheduled method; And when the onTrigger method is called, extracting and converting the variables through the expression language grammar, and obtaining the preprocessed Record. In some embodiments, the mapping the text content of the CSV file to a Record includes: And mapping the text content of the CSV file into Record records by adopting the Record readers of the Nifi clusters. In some embodiments, the mapping the text content of the CSV file to a Record using the Nifi clusters of Record readers includes: And mapping the text content of the CSV file according to Avro format fields defined in the Record reader to obtain the Record. The embodiment of the application also provides a data migration device which is applied to the service processing units in the Nifi clusters and comprises a receiving module, a first processing module and a second processing module, The receiving module is used for receiving CSV files read by other processing units of the Nifi clusters; The first processing module is used for mapping the text content of the CSV file into Record records; and the second processing module is used for preprocessing the Record to obtain a preprocessed Record, and storing the preprocessed Record into a database. The embodiment of the application also provides an electronic device comprising a processor and a memory for storing a computer program capable of running on the processor, wherein, The processor is configured to run the computer program to perform any one of the data migration methods described above. The embodiment of the application also provides a computer storage medium, on which a computer program is stored, which when executed by a processor implements any one of the above data migration methods. It can be seen that, in the embodiment of the present application, for the scenario of implementing data migration of a CSV file in the Nifi cluster, by redesigning the processing unit in the Nifi cluster, a service processing unit is utilized to implement the processes of text content reading, data mapping, data preprocessing and data warehousing, so that a shorter path of the data migration process is achieved, the number of times of generation of flowfie is reduced, the resource consumption of disk reading and writing and network transmission is reduced, and the data migration efficiency of the CSV file is improved. Drawings FIG. 1 is a flow chart of data migration of CSV files in Nifi clusters in the related art