CN-121117052-B - Data real-time processing method, system, terminal equipment and storage medium

CN121117052BCN 121117052 BCN121117052 BCN 121117052BCN-121117052-B

Abstract

The application relates to the technical field of data processing, and discloses a data real-time processing method, a system, terminal equipment and a storage medium, wherein the data real-time processing method comprises the steps of acquiring calculation data of different calculation tasks from a service database; the method comprises the steps of obtaining a data calculation rule updated in real time, checking the data calculation rule updated in real time, processing calculation data according to the data calculation rule updated in real time under the condition that the data calculation rule updated in real time passes the check, obtaining a calculation result, and writing the calculation result into a data lake. The application introduces a dynamic data calculation rule mechanism, improves the flexibility and adaptability of calculation tasks, and changes the problems that the traditional data calculation rule is solidified and cannot be dynamically adjusted. According to the method, the solidified data calculation rule codes are not required to be modified, the restarting and redeployment of the calculation tasks after the code modification are avoided, the continuity and the efficiency of task calculation are ensured, and the change of the data calculation requirement can be flexibly dealt with.

Inventors

Li Zunjian
Zhao Die

Assignees

天津金城银行股份有限公司

Dates

Publication Date: 20260505
Application Date: 20251111

Claims (8)

1. A method for real-time processing of data, comprising: Acquiring calculation data of different calculation tasks from a service database; Under the condition of determining the update of the data calculation rule of the calculation task, acquiring the data calculation rule updated in real time, and processing the calculation data stored temporarily in the memory according to the data calculation rule which is not updated; The data calculation rule refers to script programs describing how calculation data in different calculation tasks are processed, converted and calculated in the calculation tasks; under the condition that the verification of the real-time updated data calculation rule is passed, processing the calculation data according to the real-time updated data calculation rule to obtain a calculation result; The processing the calculation data according to the real-time updated data calculation rule to obtain a calculation result comprises the following steps: After the calculation data temporarily stored in the memory are processed according to the data calculation rule which is not updated, acquiring real-time calculation data, and processing the real-time calculation data according to the data calculation rule which is updated in real time; Writing the calculation result into a data lake; different computing tasks correspond to different data computing rules, and the method further comprises: Modifying the mark of the real-time updated data calculation rule under the condition that the verification of the real-time updated data calculation rule is passed, so as to determine the update of the data calculation rule of the calculation task under the condition that the mark change is detected; the flag of the data calculation rule is a status identifier indicating whether the current data calculation rule has been updated and passed the verification.
2. The method for real-time processing of data according to claim 1, wherein the acquiring the calculation data of different calculation tasks from the service database comprises: collecting the calculation data in a plurality of data tables of the service database in real time through a streaming data collection operator; And converting each calculation data into a memory storage format so as to temporarily store the calculation data in the memory storage format in the memory.
3. The method for real-time processing of data according to claim 1, wherein the acquiring the real-time updated data calculation rule and performing the verification processing on the real-time updated data calculation rule includes: The data calculation rule updated in real time is read from a dynamic rule database at regular time, and grammar check is carried out on the data calculation rule updated in real time to obtain a grammar check result; And determining whether the real-time updated data calculation rule passes the verification according to the grammar verification result.
4. The method for real-time processing of data according to claim 1, further comprising: Performing abnormal alarm under the condition that the real-time updated data calculation rule is not verified; Storing the real-time updated data calculation rules which are not checked to pass in an audit analysis database so as to record the real-time updated data calculation rules which are not checked to pass.
5. A method for real-time processing of data according to claim 3, wherein said determining whether said real-time updated data calculation rule passes verification based on said syntax checking result comprises: under the condition that the grammar checking result is wrong, determining that the real-time updated data calculation rule check fails; and under the condition that the grammar checking result is correct, determining that the real-time updated data calculation rule passes the check.
6. A data real-time processing system, comprising: the data acquisition module is used for acquiring calculation data from the service database; The rule processing module is used for acquiring a data calculation rule updated in real time and checking the data calculation rule updated in real time, wherein the data calculation rule refers to script programs describing how calculation data in different calculation tasks are processed, converted and calculated in the calculation tasks; The rule processing module is further configured to modify a flag of the real-time updated data calculation rule under a condition that the real-time updated data calculation rule passes verification, so as to determine that the data calculation rule of the calculation task is updated under a condition that the flag is detected to change; The flag of the data calculation rule is a status identifier for indicating whether the current data calculation rule has been updated and passed the verification; The data processing module is used for processing the calculation data stored temporarily in the memory according to the data calculation rule which is not updated under the condition that the data calculation rule of the calculation task is updated, processing the calculation data according to the data calculation rule which is updated in real time under the condition that the data calculation rule which is updated in real time is checked to be passed, obtaining a calculation result, and writing the calculation result into a data lake; The processing the calculation data according to the real-time updated data calculation rule to obtain a calculation result comprises the following steps: and after the calculation data temporarily stored in the memory are processed according to the data calculation rule which is not updated, acquiring real-time calculation data, and processing the real-time calculation data according to the data calculation rule which is updated in real time.
7. A terminal device, characterized in that it comprises a processor and a memory, the memory storing a computer program, the processor being adapted to execute the computer program to implement the data real-time processing method according to any one of claims 1-5.
8. A computer-readable storage medium, characterized in that it stores a computer program which, when executed on a processor, implements the data real-time processing method according to any one of claims 1-5.

Description

Data real-time processing method, system, terminal equipment and storage medium Technical Field The present application relates to the field of data processing technologies, and in particular, to a method, a system, a terminal device, and a storage medium for processing data in real time. Background With the continuous development of big data and real-time computing technology, streaming data processing plays an increasingly important role in financial scenes. Traditional real-time data computation is usually developed and deployed in a mode of customizing data tasks, namely, a developer writes computation logic codes according to specific business requirements, and packages the codes into an independent task module for deployment and operation. The traditional mode data task calculation codes are solidified and cannot be dynamically adjusted, and after the codes are modified, the calculation tasks are restarted, the task is redeployed, the operation and maintenance are manually intervened, and the task calculation efficiency is affected. Disclosure of Invention In view of the above, the embodiments of the present application provide a method, a system, a terminal device, and a storage medium for processing data in real time, which can effectively solve the problems of solidification of data task computing codes, influence on task computing efficiency, and the like. In a first aspect, an embodiment of the present application provides a method for processing data in real time, including: Acquiring calculation data of different calculation tasks from a service database; Acquiring a data calculation rule updated in real time, and checking the data calculation rule updated in real time; under the condition that the verification of the real-time updated data calculation rule is passed, processing the calculation data according to the real-time updated data calculation rule to obtain a calculation result; Writing the calculation result into a data lake. In a first possible embodiment of the first aspect, different computing tasks correspond to different data computing rules, the method further comprising: And modifying the mark of the real-time updated data calculation rule under the condition that the verification of the real-time updated data calculation rule passes, so as to determine the data calculation rule update of the calculation task under the condition that the mark change is detected. In a second possible embodiment of the first aspect, the obtaining the calculation data of the different calculation tasks from the service database includes: collecting the calculation data in a plurality of data tables of the service database in real time through a streaming data collection operator; And converting each calculation data into a memory storage format so as to temporarily store the calculation data in the memory storage format in the memory. In a third possible embodiment of the first aspect, further comprising: Under the condition of determining the updating of the data calculation rule of the calculation task, acquiring the data calculation rule updated in real time, and processing the calculation data temporarily stored in the memory according to the data calculation rule which is not updated; The processing the calculation data according to the real-time updated data calculation rule to obtain a calculation result comprises the following steps: and after the calculation data temporarily stored in the memory are processed according to the data calculation rule which is not updated, acquiring real-time calculation data, and processing the real-time calculation data according to the data calculation rule which is updated in real time. In a fourth possible embodiment of the first aspect, the acquiring a data calculation rule updated in real time, and performing a verification process on the data calculation rule updated in real time includes: The data calculation rule updated in real time is read from a dynamic rule database at regular time, and grammar check is carried out on the data calculation rule updated in real time to obtain a grammar check result; And determining whether the real-time updated data calculation rule passes the verification according to the grammar verification result. In a fifth possible embodiment of the first aspect, further comprising: Performing abnormal alarm under the condition that the real-time updated data calculation rule is not verified; Storing the real-time updated data calculation rules which are not checked to pass in an audit analysis database so as to record the real-time updated data calculation rules which are not checked to pass. In a sixth possible embodiment of the first aspect, the determining, according to the syntax checking result, whether the real-time updated data calculation rule passes a check includes: under the condition that the grammar checking result is wrong, determining that the real-time updated data calculation rule check fails; and under the