Search

CN-121455929-B - Data migration method for black box big data service system

CN121455929BCN 121455929 BCN121455929 BCN 121455929BCN-121455929-B

Abstract

The invention provides a data migration method for a black box big data service system, which relates to the technical field of financial and scientific data processing, and comprises the steps of classifying and confirming migration ranges of tables in the black box big data service system according to the activity information of a database layer table of the black box big data service system; the method comprises the steps of deploying a trigger on a database layer of a black box big data service system, tracking real-time change of a database caused by transaction through the trigger to obtain a tracking result, analyzing association relations and data distribution rules among service tables in the black box big data service system based on the tracking result, and carrying out statistical analysis on service table fields of the black box big data service system according to the data distribution rules to obtain field statistical characteristics. The invention can simultaneously ensure the integrity and the accuracy of data migration and reduce the switching production risk of the black box big data service system and the target system.

Inventors

  • ZHU YUHANG
  • WANG PENG

Assignees

  • 上海银行股份有限公司

Dates

Publication Date
20260508
Application Date
20260105

Claims (7)

  1. 1. A data migration method for a black box big data service system, the method comprising: Classifying and confirming migration ranges of tables in the black box big data service system according to the database layer table activity information of the black box big data service system, comprising the following steps: Inquiring a database system table of the black box big data service system, and acquiring the last data updating time of all tables; Based on the last data updating time, dividing tables without data change in five years into static tables, dividing tables without data change in one year into cold tables, and dividing tables with data change in one year into hot tables; performing parameter table identification and updating check on the static table, and determining a non-parameter table without updating as a non-migration table; performing offline analysis on the cold table, removing the offline table, and performing migration necessity confirmation on the cold table which is not offline; the migration confirmation of the hot table is completed by checking the hot table one by one, and the confirmation results of the static table, the cold table and the hot table are comprehensively confirmed, so that the migration range confirmation is finally completed; the method for analyzing the association relation and the data distribution rule among the business tables in the black box big data business system based on the tracking result comprises the following steps: Creating a trigger for tracking data change aiming at the service table at a database layer of the black box big data service system; Capturing a data operation record on the service table through a trigger to form an original tracking log; Analyzing and cleaning an original tracking log, and extracting a structured tracking record containing an operation type, a table name, a key field value and an operation time stamp; Based on the structured tracking record, identifying a plurality of business tables which are cooperatively changed in business transaction by carrying out association analysis according to a time sequence and a transaction identifier so as to construct association relations among the business tables; Analyzing field value change frequency and distribution in the structured tracking record, and statistically analyzing data distribution rules under different service scenes by combining the established association relation; according to the data distribution rule, carrying out statistical analysis on the business table fields of the black box big data business system to obtain field statistical characteristics; According to the association relation and the data distribution rule and referring to field statistical characteristics, converting unstructured data in an XML format in a black box big data service system into structured data; according to the confirmed migration range, the association relation and the converted structured data, a migration intermediate table for connecting the black box big data service system and the target system is designed; Aligning field-level meaning and usage between the black box big data service system and the target system through a designed migration intermediate table to obtain a field alignment result; Based on the designed migration intermediate table and field alignment result, executing migration full-period data check from the black box big data service system to the target system, and generating a check report; in the reconstruction project of the target system, the design of the migration intermediate table, the field alignment result and the check report are cooperated with the application development flow of the target system to support the function test, the data verification and the system switching of the target system.
  2. 2. The data migration method for a black box big data service system according to claim 1, wherein the performing statistical analysis on the service table field of the black box big data service system according to the data distribution rule to obtain a field statistical feature includes: Based on a data distribution rule obtained by statistical analysis, identifying a numerical field and a classification identification field in the service table field; Aiming at the identified numerical field, analyzing the numerical range, the data density and the change trend of the numerical field to form numerical distribution characteristics; Aiming at the identified classification identification field, analyzing a value set, frequency of occurrence and stability of the classification identification field to form enumeration distribution characteristics; combining data distribution rules under different service scenes, performing scene verification and correction on the numerical distribution characteristics and the enumeration distribution characteristics, and generating a scene verification result; integrating the numerical distribution characteristics, the enumeration distribution characteristics and the scene verification results to generate field statistical characteristics.
  3. 3. The data migration method for the black-box big data service system according to claim 2, wherein converting unstructured data in XML format in the black-box big data service system into structured data according to the association relationship and the data distribution rule and referring to the field statistical feature, comprises: based on the established association relation, analyzing nesting level and reference relation of data nodes in the unstructured data in XML format to form a node relation model; Determining rules for splitting and mapping XML data nodes according to the node relation model, the data distribution rules and the field statistical characteristics; According to the determined splitting and mapping rule, carrying out hierarchical analysis and value extraction on XML format data by adopting a multidimensional splitting method, and generating an atomic data unit corresponding to the node in the node relation model; According to the logical correspondence between the atomic data units and the node relation model, reorganizing and mapping the atomic data units into records of a target structured table to generate a record set; And (3) carrying out data integrity verification on the target structured table based on the record set to confirm the logical consistency of the target structured table and XML source data, and finally completing the conversion of the structured data.
  4. 4. The method for migrating data of black box big data service system according to claim 3, wherein the migration range comprises a static parameter table, a cold table and a hot table for migrating and confirming, wherein the association relation and the converted structured data are used for designing a migration intermediate table for connecting the black box big data service system and a target system, and the method comprises the following steps: determining a source service table set needing to be incorporated into a migration intermediate table based on the confirmed migration range; carrying out service logic association analysis on the source service list set by combining the constructed association relation to define a core service module and a data theme which are born by the migration intermediate list; based on the defined core service module and data theme, and according to the converted structured data, referring to field statistics characteristics, designing a table structure of a migration intermediate table, wherein the table structure of the migration intermediate table covers all source service table fields and meets service relevance checking requirements; according to the design of the table structure of the target system, a mapping and splitting rule of the migration intermediate table to one or more target tables of the target system is established, wherein the table structure of the target system can be a preliminary design version and is adjusted in an iterative manner in the migration process; Based on the mapping and splitting rules, verifying and optimizing the table structure of the designed migration intermediate table to ensure that the migration intermediate table can meet the requirements of accurate and efficient data migration from the black box big data service system to the target system, and finally completing the design of the migration intermediate table.
  5. 5. The data migration method for a black-box big data service system according to claim 4, wherein the field-level meaning and usage alignment between the black-box big data service system and the target system is performed through a designed migration intermediate table, and a field alignment result is obtained, which includes: Identifying a source field from the black box big data service system and a target field corresponding to the target system in the migration intermediate table based on the design of the final migration intermediate table; Aiming at a common field with the same meaning and name in a source field and a target field, comparing and analyzing service meanings and usage rules of the common field in a black box big data service system and a target system to obtain a first field alignment result; Aiming at the unique and necessary technical fields in the target fields, designing and realizing the generation logic of the values of the technical fields in the migration intermediate table according to the technical rules provided by the target system, and obtaining a second-type field alignment result; aiming at the unique and necessary service fields in the target fields, carrying out data complement or mapping rule confirmation by cooperative service personnel to obtain a third type field alignment result; Aiming at the fields which are unique in the source field and missing in the target field, the cooperative service and the developer confirm whether the missing fields need to be supplemented to the target system or not, and a supplementing scheme is formulated to obtain a fourth type field alignment result; Integrating the first field alignment result, the second field alignment result, the third field alignment result and the fourth field alignment result to generate a complete field alignment result.
  6. 6. The data migration method for a black-box large data service system according to claim 5, wherein performing migration full-period data inspection from the black-box large data service system to the target system based on the designed migration intermediate table and field alignment result, generating an inspection report, comprises: Based on the design of the finally completed migration intermediate table and the complete field alignment result, a full-period checking strategy comprising before migration, during migration and after migration is formulated; Checking the consistency of the table structures of the infrastructure, the black box big data service system and the target system related to migration and the version of the migration program according to the full-period checking strategy, ensuring that the migration starting condition is met, and generating a checking result before migration; According to the complete field alignment result and the design of the migration intermediate table, in the process of transferring data from the black box big data service system to the target system through the migration intermediate table, carrying out staged record number comparison and service rule verification to generate a check result during migration; after the data is completely migrated into the target system, carrying out total quantity check, multidimensional summarization check and accounting check on the migrated data in the target system according to the complete field alignment result, and generating a post-migration check result; integrating the checking result before migration, the checking result during migration and the checking result after migration to generate a checking report for comprehensively reflecting the accuracy and the integrity state of data migration.
  7. 7. The method for migrating data in a black-box big data service system according to claim 6, wherein in the reconfiguration project of the target system, the design of the migration intermediate table, the field alignment result and the check report are cooperated with the application development flow of the target system to support the function test, the data verification and the system switching of the target system, comprising: In the application development flow of the target system, an integration strategy which is cooperated with the application development flow of the target system is formulated based on the design of the finally completed migration intermediate table, the complete field alignment result and the check report; In the demand analysis stage of the target system, input and verification basis is provided for the functional demand and the data model design of the target system based on the design of the finally completed migration intermediate table and the complete field alignment result; In the design and development stage of the target system, according to the complete field alignment result and the check report, the alignment of the target system table structure and the business logic and the migration data specification is completed cooperatively; In the testing stage of the target system, according to the design and check report of the migration intermediate table which is finally completed, supporting the functional test, data verification and double-shot comparison test of the target system; And in the online switching stage of the target system, supporting the target system to finish the service acceptance of the migration data according to the check report and the aligned migration data, and realizing the smooth switching between the black box big data service system and the target system.

Description

Data migration method for black box big data service system Technical Field The invention relates to the technical field of financial and scientific data processing, in particular to a data migration method for a black box big data service system. Background In the system upgrading and updating process in the field of financial science and technology, data migration is a core link for guaranteeing business continuity of new and old systems. The black box big data service system has the technical locking characteristic that the application layer codes and interfaces are highly packaged, and the technical defect that the association relation and the data distribution rule between service tables cannot be directly obtained is common, and the defect becomes a key bottleneck for restricting the cross-system data migration quality. On one hand, due to the lack of clear cognition on the association relation of the service tables, the existing migration scheme is difficult to accurately define the migration range, namely the service attributes and migration necessity of a hot table, a cold table and a static table cannot be effectively distinguished, the problem that key service data is missed or redundant waste data is wrongly migrated easily occurs, the integrity of data migration is directly damaged, and the continuity of the service life cycle of a new system is further affected. On the other hand, due to the unknown data distribution rule, a large amount of unstructured data in an XML format in a black box system is difficult to effectively analyze, so that the nested logic and mapping rule of data nodes cannot be clarified, the numerical characteristics, enumeration range and abnormal data conditions of fields cannot be identified, the conversion of the unstructured data to the structured data is blocked, the meaning and usage of fields between new and old systems cannot be aligned accurately, the accuracy of migration data is finally insufficient, and the function realization and service acceptance of a target system are seriously affected. In summary, the existing data migration method cannot provide an effective solution for the technical blocking defect of the black box system, and cannot meet the dual requirements of migration integrity and accuracy at the same time. Disclosure of Invention The invention aims to solve the technical problem of providing a data migration method for a black box big data service system, which can ensure the integrity and the accuracy of migration data, reduce the switching production risk of the black box big data service system and a target system, and provide general and reliable technical support for upgrading and updating the black box system in the field of financial science and technology. In order to solve the technical problems, the technical scheme of the invention is as follows: A data migration method for a black box big data service system, the method comprising: Classifying and confirming migration ranges of tables in the black box big data service system according to the database layer table activity information of the black box big data service system; disposing a trigger on a database layer of the black box big data service system, tracking real-time change of the database caused by transaction through the trigger to obtain a tracking result; according to the data distribution rule, carrying out statistical analysis on the business table fields of the black box big data business system to obtain field statistical characteristics; According to the association relation and the data distribution rule and referring to field statistical characteristics, converting unstructured data in an XML format in a black box big data service system into structured data; according to the confirmed migration range, the association relation and the converted structured data, a migration intermediate table for connecting the black box big data service system and the target system is designed; Aligning field-level meaning and usage between the black box big data service system and the target system through a designed migration intermediate table to obtain a field alignment result; Based on the designed migration intermediate table and field alignment result, executing migration full-period data check from the black box big data service system to the target system, and generating a check report; in the reconstruction project of the target system, the design of the migration intermediate table, the field alignment result and the check report are cooperated with the application development flow of the target system to support the function test, the data verification and the system switching of the target system. The scheme of the invention at least comprises the following beneficial effects: The technical means of table classification and confirmation of migration range, data association and distribution rule analysis by a database layer trigger, conversion of XML unstructured data b