CN-115203339-B - Multi-data source integration method, device, computer equipment and storage medium

CN115203339BCN 115203339 BCN115203339 BCN 115203339BCN-115203339-B

Abstract

The embodiment of the application provides a multi-data source integration method, a device, computer equipment and a storage medium, wherein the method acquires a data source integration request of a data warehouse, the data source integration request comprises business data to be processed and integration parameters, the data source integration request is verified through a service unit, under the condition that the verification is passed, the business data to be processed is analyzed through an analysis unit and the integration parameters, an integration index and a data source type are determined, a target component matched with the data source type is determined from a plurality of components of an engine unit, and the business data to be processed is integrated through the target component and the service unit according to the integration index to obtain standard integration data, so that the standardization and unified processing of the data of different data sources are realized, the integration efficiency of the multi-data source is improved, the maintainability and the expansibility of the data warehouse are improved, and the rapid access and consumption capability of the data are improved.

Inventors

NIU YUCHAO

Assignees

平安银行股份有限公司

Dates

Publication Date: 20260512
Application Date: 20220624

Claims (8)

1. A multiple data source integration method, applied to a cloud platform server, the cloud platform server comprising a data warehouse configured as a service unit for checksum aggregation of data, a parsing unit for parsing the data, and an engine unit for routing to a target component, and the engine unit comprising a plurality of components, each component corresponding to a data source, each component comprising a structured query statement corresponding to a respective field, the method comprising: acquiring a data source integration request for the data warehouse, wherein the data source integration request comprises business data to be processed and integration parameters; Checking the data source integration request through the service unit; under the condition that verification is passed, analyzing the business data to be processed through the analysis unit and the integration parameters, and determining an integration index and a data source type; determining a target component matching the data source type from a plurality of components of the engine unit; Integrating the service data to be processed through the target component and the service unit according to the integration index, wherein the service unit acquires a field contained in the service data to be processed, acquires the same field contained in the service data to be processed from the field, and combines a plurality of service data to be processed according to the same field to obtain initial integration data; When the data source type of the service data to be processed is detected to be the newly added data source type, a data model matched with the newly added data source type is built in the engine unit; determining a script of the data model; And executing the script to obtain the target component matched with the newly added data source type.
2. The multi-data-source integration method of claim 1, wherein the service unit includes at least one of a plurality of standard data-source integration requests and a check expression corresponding to each of the standard data-source integration requests, the check expression indicating a check rule of the data-source integration request; the verifying, by the service unit, the data source integration request includes: Comparing the data source integration request with each standard data source integration request, judging whether the verification passes or not according to the comparison result, or, And checking the data source integration request by using the check expression, and judging whether the check is passed.
3. The multi-data source integration method of claim 1, wherein the parsing unit comprises a database address identification tool and an index parsing logic expression, wherein the index parsing logic expression is used for indicating a parsing rule of the integration parameter; And under the condition that the verification is passed, analyzing the service data to be processed through the analysis unit and the integration parameter, and determining an integration index and a data source type, wherein the method comprises the following steps of: identifying the service data to be processed by utilizing the database address identification tool to obtain the data source type; and logically analyzing the definition of the integration parameter through the index analysis logic expression to obtain the integration index.
4. The multiple data source integration method of claim 3, wherein the determining a target component from among the plurality of components of the engine unit that matches the data source type comprises: and inputting the data source type, the service data to be processed and the integration index into a route matching model to obtain the target component.
5. The multiple data source integration method of claim 4, further comprising: If the existence of the components in the engine unit is detected, determining that the components are components to be deleted if the components are not components of the target components within a preset time period; and deleting the component to be deleted from the engine unit.
6. A multiple data source integration apparatus applied to a cloud platform server including a data repository configured as a service unit for checksum aggregation of data, a parsing unit for parsing the data, and an engine unit for routing to a target component, and the engine unit includes a plurality of components, each component corresponding to a data source, each component including a structured query statement corresponding to a respective field, the apparatus comprising: the receiving module is used for acquiring a data source integration request of the data warehouse, wherein the data source integration request comprises business data to be processed and integration parameters; The verification module is used for verifying the data source integration request through the service unit; The analysis module is used for analyzing the business data to be processed through the analysis unit and the integration parameters under the condition that the verification is passed, and determining an integration index and a data source type; A determining module for determining a target component matching the data source type from a plurality of components of the engine unit; The system comprises an integration module, an engine unit, a data model, a script and a target component, wherein the integration module is used for integrating business data to be processed through the target component and the service unit according to the integration index, acquiring fields contained in the business data to be processed through the service unit, acquiring the same fields contained in the business data to be processed from the fields, carrying out association and combination on a plurality of business data to be processed according to the same fields to obtain initial integration data, taking the initial integration data as input parameters of a structured query statement, carrying out standardized processing on the initial integration data through the target component to obtain standard integration data, constructing a data model matched with the newly added data source type in the engine unit when the data source type of the business data to be processed is detected to be the newly added data source type, determining the script of the data model, and executing the script to obtain the target component matched with the newly added data source type.
7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the multiple data source integration method according to any one of claims 1 to 5 when the computer program is executed.
8. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the steps of the multiple data source integration method of any one of claims 1 to 5.

Description

Multi-data source integration method, device, computer equipment and storage medium Technical Field The present application relates to the field of network databases, and in particular, to a method and apparatus for integrating multiple data sources, a computer device, and a storage medium. Background With the development of information technology, the data volume of enterprises or units is increased at a high speed, taking performance index data of bank credit cards as an example, when a credit card OLAP system is built, multiple data sources such as Oracle data, SQLServer data, API data and the like are often needed to be docked, and the requirements of the OLAP system for quickly docking new data sources and quickly consuming data are difficult to be met due to different storage specifications of different data sources, so that it is necessary to provide an efficient multi-data source data integration method to improve the processing efficiency of multi-data source data. Content of the application The embodiment of the application provides a multi-data source integration method, a multi-data source integration device, computer equipment and a storage medium, so as to solve the technical problem of low processing performance of multi-data source data caused by different storage specifications of different data sources. In one aspect, the present application provides a multi-data source integration method applied to a cloud platform server, the cloud platform server including a data warehouse, the data warehouse being configured as a service unit for checking and aggregating data, an analysis unit for analyzing the data, and an engine unit for routing to a target component, and the engine unit including a plurality of components, each component corresponding to a data source, the method comprising: acquiring a data source integration request for the data warehouse, wherein the data source integration request comprises business data to be processed and integration parameters; Checking the data source integration request through the service unit; under the condition that verification is passed, analyzing the business data to be processed through the analysis unit and the integration parameters, and determining an integration index and a data source type; determining a target component matching the data source type from a plurality of components of the engine unit; and integrating the business data to be processed through the target component and the service unit according to the integration index to obtain standard integrated data. In one aspect, the present application provides a multiple data source integration apparatus applied to a cloud platform server, the cloud platform server including a data warehouse configured as a service unit for checking and aggregating data, an parsing unit for parsing the data, and an engine unit for routing to a target component, and the engine unit includes a plurality of components, each component corresponding to one data source, the apparatus comprising: the receiving module is used for acquiring a data source integration request of the data warehouse, wherein the data source integration request comprises business data to be processed and integration parameters; The verification module is used for verifying the data source integration request through the service unit; The analysis module is used for analyzing the business data to be processed through the analysis unit and the integration parameters under the condition that the verification is passed, and determining an integration index and a data source type; A determining module for determining a target component matching the data source type from a plurality of components of the engine unit; And the integration module is used for integrating the service data to be processed through the target component and the service unit according to the integration index to obtain standard integration data. In one aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps in the multi-data source integration method described above. In one aspect, the present application provides a computer readable medium storing a computer program which, when executed by a processor, causes the processor to perform the steps in the multi-data source integration method described above. The embodiment of the application provides a multi-data source integration method, which is characterized in that a data source integration request of a data warehouse is obtained, the data source integration request contains business data to be processed and integration parameters, the data source integration request is verified through a service unit, under the condition that the verification is passed, the business data to be processed is analyzed through an analysis unit and the integration parameters, an inte