CN-115827601-B - Data processing method, device, equipment and storage medium

CN115827601BCN 115827601 BCN115827601 BCN 115827601BCN-115827601-B

Abstract

The application provides a data processing method, a data processing device, data processing equipment and a storage medium. The method comprises the steps of obtaining metadata to be processed, wherein the metadata comprise extended attribute metadata and general attribute metadata, obtaining a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata, carrying out format conversion on the database storage type, the storage structure version number, the extended attribute name and the extended attribute value by adopting a preset meta model to obtain converted extended attribute metadata, and storing the converted extended attribute metadata and the general attribute metadata. The method of the application can convert the formats of the metadata of different databases by setting the universal meta model, is convenient for unified management of the metadata, not only effectively reduces the storage space, but also is beneficial to providing the data transmission efficiency.

Inventors

JIA YUWU
SONG YULUN
LI DAZHONG
TAN CHENGZHONG
QIN HONGWEI
Wu Yiliqi

Assignees

中国联合网络通信集团有限公司
联通数字科技有限公司

Dates

Publication Date: 20260505
Application Date: 20221214

Claims (9)

1. A method of data processing, the method comprising: acquiring metadata to be processed, wherein the metadata comprises extended attribute metadata and general attribute metadata; Acquiring a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata; performing format conversion on the database storage type, the storage structure version number, the extended attribute name and the extended attribute value by adopting a preset meta model to obtain converted extended attribute metadata; storing the converted extended attribute metadata and the generic attribute metadata; The format conversion is performed on the database storage type, the storage structure version number, the extended attribute name and the extended attribute value by adopting a preset meta model to obtain converted extended attribute metadata, and the method comprises the following steps: acquiring a first mapping relation between a preset database storage type and a standard code, a second mapping relation between a preset storage structure version number and the standard code, a third mapping relation between a preset extended attribute name and the standard code and a fourth mapping relation between a preset extended attribute value and the standard code in a preset meta model; Respectively carrying out format conversion on the database storage type and the storage structure version number based on the first mapping relation and the second mapping relation to obtain a converted database storage type and a converted storage structure version number; performing format conversion on the extended attribute names based on the third mapping relation to obtain converted extended attribute names, and determining first extended attribute metadata based on the converted extended attribute names; Performing format conversion on the extended attribute value based on the fourth mapping relation to obtain a converted extended attribute value, and determining second extended attribute metadata based on the converted extended attribute value; and sequencing the converted database storage type, the converted storage structure version number, the first extended attribute metadata and the second extended attribute metadata according to a preset sequence to obtain the converted extended attribute metadata.
2. The method of claim 1, wherein the determining the first extended attribute metadata based on the converted extended attribute name comprises: acquiring the length corresponding to the converted extended attribute name and the attribute type identifier of the extended attribute name; and determining the attribute type identification of the extended attribute name, the length and the converted extended attribute name as the first extended attribute metadata.
3. The method of claim 1, wherein the determining second extended attribute metadata based on the converted extended attribute value comprises: acquiring the length corresponding to the converted extended attribute value and the attribute type identifier of the extended attribute value; and determining the attribute type identification of the extended attribute value, the length and the converted extended attribute value as the second extended attribute metadata.
4. A method according to any one of claims 1 to 3, wherein said storing said converted extended attribute metadata and said generic attribute metadata comprises: Encrypting the converted extended attribute metadata by adopting a preset symmetric encryption algorithm to obtain encrypted extended attribute metadata; And storing the encrypted extended attribute metadata and the general attribute metadata into a preset database.
5. A method of data processing, the method comprising: Sending a metadata query request to a corresponding server, so that the server feeds back extended attribute metadata corresponding to the request and corresponding general attribute metadata, wherein the extended attribute metadata are obtained by converting a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata by adopting a preset meta model; receiving the corresponding extended attribute metadata and the corresponding general attribute metadata sent by the server; Performing format conversion on the corresponding extended attribute metadata to obtain original extended attribute metadata; displaying the original extended attribute metadata and the corresponding general attribute metadata; The format conversion of the corresponding extended attribute metadata is performed to obtain original extended attribute metadata, which includes: acquiring a first mapping relation between a preset database storage type and a standard code, a second mapping relation between a preset storage structure version number and the standard code, a third mapping relation between a preset extended attribute name and the standard code and a fourth mapping relation between a preset extended attribute value and the standard code in a preset meta model; Based on the first mapping relation and the second mapping relation, respectively performing format conversion on the database storage type and the storage structure version number in the corresponding extended attribute metadata to obtain an original database storage type and an original storage structure version number; converting the extended attribute names in the corresponding extended attribute metadata based on the third mapping relation to obtain original extended attribute names; Converting the extended attribute value in the corresponding extended attribute metadata based on the fourth mapping relation to obtain an original extended attribute value; And obtaining original extended attribute metadata based on the original database storage type, the original storage structure version number, the original extended attribute name and the original extended attribute value.
6. A data processing apparatus, the apparatus comprising: The device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring metadata to be processed, and the metadata to be processed comprises extended attribute metadata and general attribute metadata; the acquisition unit is also used for acquiring the storage type, the storage structure version number, the extended attribute name and the extended attribute value of the database corresponding to the extended attribute metadata; The first processing unit is used for carrying out format conversion on the database storage type, the storage structure version number, the extended attribute name and the extended attribute value by adopting a preset meta model to obtain converted extended attribute metadata; a storage unit configured to store the converted extended attribute metadata and the generic attribute metadata; Wherein, the first processing unit is specifically configured to: acquiring a first mapping relation between a preset database storage type and a standard code, a second mapping relation between a preset storage structure version number and the standard code, a third mapping relation between a preset extended attribute name and the standard code and a fourth mapping relation between a preset extended attribute value and the standard code in a preset meta model; Respectively carrying out format conversion on the database storage type and the storage structure version number based on the first mapping relation and the second mapping relation to obtain a converted database storage type and a converted storage structure version number; performing format conversion on the extended attribute names based on the third mapping relation to obtain converted extended attribute names, and determining first extended attribute metadata based on the converted extended attribute names; Performing format conversion on the extended attribute value based on the fourth mapping relation to obtain a converted extended attribute value, and determining second extended attribute metadata based on the converted extended attribute value; and sequencing the converted database storage type, the converted storage structure version number, the first extended attribute metadata and the second extended attribute metadata according to a preset sequence to obtain the converted extended attribute metadata.
7. A data processing apparatus, the apparatus comprising: The receiving and transmitting unit is used for transmitting a metadata query request to a corresponding server so that the server feeds back extended attribute metadata corresponding to the request and corresponding general attribute metadata, wherein the extended attribute metadata are extended attribute metadata obtained by converting a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata by adopting a preset meta model; The receiving and transmitting unit is also used for receiving the corresponding extended attribute metadata and the corresponding general attribute metadata sent by the server; the second processing unit is used for carrying out format conversion on the corresponding extended attribute metadata to obtain original extended attribute metadata; the display unit is used for displaying the original extended attribute metadata and the corresponding general attribute metadata; the second processing unit is specifically configured to: acquiring a first mapping relation between a preset database storage type and a standard code, a second mapping relation between a preset storage structure version number and the standard code, a third mapping relation between a preset extended attribute name and the standard code and a fourth mapping relation between a preset extended attribute value and the standard code in a preset meta model; Based on the first mapping relation and the second mapping relation, respectively performing format conversion on the database storage type and the storage structure version number in the corresponding extended attribute metadata to obtain an original database storage type and an original storage structure version number; converting the extended attribute names in the corresponding extended attribute metadata based on the third mapping relation to obtain original extended attribute names; Converting the extended attribute value in the corresponding extended attribute metadata based on the fourth mapping relation to obtain an original extended attribute value; And obtaining original extended attribute metadata based on the original database storage type, the original storage structure version number, the original extended attribute name and the original extended attribute value.
8. An electronic device includes a processor, and a memory communicatively coupled to the processor; the memory stores computer-executable instructions; the processor executing computer-executable instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 5.
9. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1 to 5.

Description

Data processing method, device, equipment and storage medium Technical Field The present application relates to the field of data management technologies, and in particular, to a data processing method, apparatus, device, and storage medium. Background Metadata is data about data, which describes, interprets, locates and makes it more convenient to retrieve, use or manage. Models in various storage databases in the data warehouse exist in respective meta-model management systems, and each storage database has its own meta-model system, such as a hive meta-model system metastore, a hbase meta-model system meta and root table, and the like. However, when data of various types of storage databases are stored and managed in a centralized manner, meta-models of various types of databases need to be increased continuously, and the meta-data cannot be managed in a unified manner effectively aiming at the existing heterogeneous storage scene. Disclosure of Invention The application provides a data processing method, a device, equipment and a storage medium, which are used for solving the problem that the conventional heterogeneous storage scene cannot effectively perform unified management on metadata. In a first aspect, the present application provides a data processing method, including: acquiring metadata to be processed, wherein the metadata comprises extended attribute metadata and general attribute metadata; Acquiring a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata; performing format conversion on the database storage type, the storage structure version number, the extended attribute name and the extended attribute value by adopting a preset meta model to obtain converted extended attribute metadata; and storing the converted extended attribute metadata and the general attribute metadata. In a second aspect, the present application further provides a data processing method, including: Sending a metadata query request to a corresponding server, so that the server feeds back extended attribute metadata corresponding to the request and corresponding general attribute metadata, wherein the extended attribute metadata are obtained by converting a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata by adopting a preset meta model; receiving the corresponding extended attribute metadata and the corresponding general attribute metadata sent by the server; Performing format conversion on the corresponding extended attribute metadata to obtain original extended attribute metadata; and displaying the original extended attribute metadata and the corresponding general attribute metadata. In a third aspect, the present application provides a data processing apparatus comprising: The device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring metadata to be processed, and the metadata to be processed comprises extended attribute metadata and general attribute metadata; the acquisition unit is also used for acquiring the storage type, the storage structure version number, the extended attribute name and the extended attribute value of the database corresponding to the extended attribute metadata; The first processing unit is used for carrying out format conversion on the database storage type, the storage structure version number, the extended attribute name and the extended attribute value by adopting a preset meta model to obtain converted extended attribute metadata; And the storage unit is used for storing the converted extended attribute metadata and the general attribute metadata. In a fourth aspect, the present application also provides a data processing apparatus, including: The receiving and transmitting unit is used for transmitting a metadata query request to a corresponding server so that the server feeds back extended attribute metadata corresponding to the request and corresponding general attribute metadata, wherein the extended attribute metadata are extended attribute metadata obtained by converting a database storage type, a storage structure version number, an extended attribute name and an extended attribute value corresponding to the extended attribute metadata by adopting a preset meta model; The receiving and transmitting unit is also used for receiving the corresponding extended attribute metadata and the corresponding general attribute metadata sent by the server; the second processing unit is used for carrying out format conversion on the corresponding extended attribute metadata to obtain original extended attribute metadata; And the display unit is used for displaying the original extended attribute metadata and the corresponding general attribute metadata. In a fifth aspect, the present applicati