CN-122019661-A - Collaborative synchronization method, system and medium for financial business data

CN122019661ACN 122019661 ACN122019661 ACN 122019661ACN-122019661-A

Abstract

The application provides a collaborative synchronization method, a collaborative synchronization system and a collaborative synchronization medium of financial service data, wherein the method comprises the steps of calling a Flink CDC component to subscribe and analyze a Binlog log of a Mysql database in real time; data acquisition is carried out on the analyzed Binlog original data through the Flink CDC component so as to obtain service related data; the business related data is cleaned and converted through the Flink CDC component to form standard data; the method comprises the steps of obtaining a source library type of standard data, writing the standard data into a specified subject area in a Kafka queue in a partitioning mode according to the source library type, subscribing and obtaining Kafka information of the Kafka queue based on Java synchronous service deployed at the downstream, and writing the data to be synchronized into target equipment in batches based on a preset mapping rule according to the content of the Kafka information as the data to be synchronized.

Inventors

SHI JIANPENG

Assignees

上海数禾信息科技有限公司

Dates

Publication Date: 20260512
Application Date: 20251226

Claims (10)

1. A collaborative synchronization method for financial business data, the method comprising: Calling a Flink CDC component to subscribe and analyze Binlog logs of the Mysql database in real time; data acquisition is carried out on the analyzed Binlog original data through the Flink CDC component so as to obtain service related data, wherein the service related data comprises full historical data and real-time change data of a post-loan service core table; The business related data is cleaned and converted through the Flink CDC component to form standard data; Acquiring a source library type of the standard data, and writing the standard data into a specified subject area in a Kafka queue in a partition mode according to the source library type; Subscribing and acquiring the Kafka message of the Kafka queue based on Java synchronous service deployed downstream, taking the content of the Kafka message as data to be synchronized, and writing the data to be synchronized into target equipment in batches based on a preset mapping rule.
2. The collaborative synchronization method of financial transaction data according to claim 1, wherein the data collection of the Binlog raw data through the link CDC component to obtain transaction related data comprises: Acquiring a log starting site according to the Binlog log; Sequentially lockless scanning the full amount of historical data of the Mysql database through the link CDC component based on the log start site, and synchronously caching real-time change data generated during scanning; And after merging the full historical data and the real-time change data together, re-entering Binlog monitoring according to the log starting site.
3. The collaborative synchronization method of financial transaction data according to claim 1, wherein the forming standard data after the cleansing transformation of the transaction related data via the Flink CDC component comprises: The Flink CDC component converts the service related data into a unified data structure, wherein the data structure is a preset structure.
4. The collaborative synchronization method of financial transaction data according to claim 1, wherein the preset mapping rules further include a mapping relationship between the source library type and a specified subject area in the Kafka queue, and wherein standard data of each same source library type is written into the same specified subject area in the Kafka queue according to the preset mapping rules.
5. The collaborative synchronization method of financial transaction data according to claim 1, wherein the subscribing and retrieving the Kafka message of the Kafka queue based on the downstream deployed Java synchronization service, retrieving data to be synchronized according to the Kafka message, comprises: Subscribing to the Kafka message in the Kafka queue after downstream deployment of the Java synchronization service; Pulling the Kafka messages in the Kafka queue through the Java synchronous service, and sequentially reading the Kafka messages; the batch writing of the data to be synchronized into the target device based on the preset mapping rule comprises the following steps: Acquiring a first data structure of the data to be synchronized, and acquiring and determining a second data structure corresponding to the data to be synchronized according to the first data structure and the preset mapping rule; converting the data to be synchronized into a second data structure to obtain target data; Calling an API of the target equipment to write the target data into the target equipment in sequence.
6. The collaborative synchronization method of financial transaction data according to claim 5, wherein the preset mapping rules include at least one of a single-table data mapping single index, a multi-table associated data mapping wide-table index, and a historical real-time data mapping sub-index store.
7. The collaborative synchronization method of claim 5, wherein during sequential writing of the target data to the target device, writing is stopped when a dynamic buffer threshold is exceeded, the dynamic buffer threshold being a maximum write value at which downtime is not available when the target device writes data.
8. The collaborative synchronization method of financial transaction data of any one of claims 1-7, wherein the preset mapping rules are configured based on a configurable rules engine.
9. A collaborative synchronization system for financial transaction data, the system comprising: the calling module is used for calling the Flink CDC component to subscribe and analyze the Binlog log of the Mysql database in real time; The data acquisition module is used for acquiring the analyzed Binlog original data through the Flink CDC component so as to acquire service related data, wherein the service related data comprises full historical data and real-time change data of a post-credit service core table; The conversion module is used for forming standard data after cleaning and converting the service related data through the Flink CDC component; The partition writing module is used for obtaining the source library type of the standard data and writing the standard data partition into a specified subject area in a Kafka queue according to the source library type; and the synchronization module is used for subscribing and acquiring the Kafka message of the Kafka queue based on Java synchronization service deployed at the downstream, taking the content of the Kafka message as data to be synchronized, and writing the data to be synchronized into target equipment in batches based on a preset mapping rule.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that the program, when executed, implements the collaborative synchronization method of financial transaction data of any one of claims 1 to 8.

Description

Collaborative synchronization method, system and medium for financial business data Technical Field The application belongs to the technical field of data management, relates to a business data synchronization method, and in particular relates to a collaborative synchronization method, a collaborative synchronization system and a collaborative synchronization medium for financial business data. Background With the rapid development of internet technology, data is in explosive growth, and enterprises have higher requirements on data processing instantaneity, query efficiency and data analysis capability. MySQL, which is a representation of a relational database, is widely used for data storage in core business systems with its stability, transactional consistency, and structured storage advantages. However, mySQL performance is difficult to meet when dealing with massive data full text retrieval, complex aggregate analysis, and high concurrency query scenarios. A distributed search Engine (ES) is constructed based on Lucene, has near real-time search, horizontal expansion capability and powerful full-text retrieval function, and can efficiently process complex queries of unstructured/semi-structured data. Therefore, service data in MySQL is synchronized to ES, a framework of 'MySQL memory core data+ES support efficient query' is formed, and a mainstream scheme for enterprises to solve data retrieval and analysis pain points is formed. However, in the process of synchronizing the data of the Mysql database to the ES, there are problems that the real-time performance and consistency are difficult to be compatible, the processing capability of the complex scene is weak, and the flexible mapping conversion capability is lacking. Disclosure of Invention The application aims to provide a collaborative synchronization method, a collaborative synchronization system and a collaborative synchronization medium for financial service data, which are used for solving the problem that in the prior art, consistency and instantaneity are difficult to be compatible when a Mysql database is synchronized to an ES, so that efficiency is low. In a first aspect, the present application provides a collaborative synchronization method for financial service data, the method comprising: Calling a Flink CDC component to subscribe and analyze Binlog logs of the Mysql database in real time; data acquisition is carried out on the analyzed Binlog original data through the Flink CDC component so as to obtain service related data, wherein the service related data comprises full historical data and real-time change data of a post-loan service core table; The business related data is cleaned and converted through the Flink CDC component to form standard data; Acquiring a source library type of the standard data, and writing the standard data into a specified subject area in a Kafka queue in a partition mode according to the source library type; Subscribing and acquiring the Kafka message of the Kafka queue based on Java synchronous service deployed downstream, taking the content of the Kafka message as data to be synchronized, and writing the data to be synchronized into target equipment in batches based on a preset mapping rule. In an implementation manner of the first aspect, the data acquisition of the Binlog raw data by the link CDC component to obtain service related data includes: Acquiring a log starting site according to the Binlog log; Sequentially lockless scanning the full amount of historical data of the Mysql database through the link CDC component based on the log start site, and synchronously caching real-time change data generated during scanning; And after merging the full historical data and the real-time change data together, re-entering Binlog monitoring according to the log starting site. In an implementation manner of the first aspect, the forming standard data after the cleaning and converting the service related data by using the link CDC component includes: The Flink CDC component converts the service related data into a unified data structure, wherein the data structure is a preset structure. In an implementation manner of the first aspect, the preset mapping rule further includes a mapping relationship between the source library type and a specified topic area in the Kafka queue, and standard data of each same source library type is written into the same specified topic area in the Kafka queue according to the preset mapping rule. In an implementation manner of the first aspect, the subscribing and acquiring, by the Java synchronization service based on the downstream deployment, a Kafka message of the Kafka queue, and acquiring data to be synchronized according to the Kafka message includes: Subscribing to the Kafka message in the Kafka queue after downstream deployment of the Java synchronization service; Pulling the Kafka messages in the Kafka queue through the Java synchronous service, and sequentially reading th