Search

CN-122020468-A - Intelligent analysis early warning and decision support system and method based on big data processing

CN122020468ACN 122020468 ACN122020468 ACN 122020468ACN-122020468-A

Abstract

The invention discloses an intelligent analysis early warning and decision support system and method based on big data processing, which relate to the technical field of big data intelligent analysis and are used for constructing and updating a dynamic knowledge graph taking a business entity as a node and a multiple relation between entities as edges in real time, carrying out context information enhancement by combining a generated unified feature vector with a neighborhood structure and a related entity state of the entity in the knowledge graph to form an enhanced feature vector, inputting the enhanced feature vector into a plurality of parallel early warning analysis models to generate comprehensive early warning signals and confidence, and realizing dynamic unification of real-time flow features and historical batch features by using a time-varying weight self-adaptive fusion algorithm, so that the situation of data processing splitting is broken, and the system simultaneously has real-time response capability and historical data support, thereby improving the comprehensiveness and the accuracy of analysis results.

Inventors

  • CHEN JUNAN

Assignees

  • 成都恒迈科技有限公司

Dates

Publication Date
20260512
Application Date
20260129

Claims (10)

  1. 1. The intelligent analysis early warning and decision support method based on big data processing is characterized by comprising the following steps: Step 1, analyzing, cleaning and converting accessed real-time stream data and batch historical data to unify time sequence feature vectors and attribute tags in a preset format; Step 2, performing windowed real-time calculation on the real-time stream data, and performing periodic depth aggregation calculation on batch historical data; step 3, constructing and updating a dynamic knowledge graph with business entities as nodes and the multiple relations among the entities as edges in real time, and carrying out context information enhancement by combining the neighborhood structure of the entities in the knowledge graph and the states of the related entities to form an enhanced feature vector; Step 4, the enhanced feature vector is input into a plurality of parallel early warning analysis models, wherein the early warning analysis models comprise an anomaly detection model, a trend prediction model and a business rule model; And 5, when the confidence level of the comprehensive early warning signal exceeds a preset threshold, taking the early warning entity as a starting point, carrying out multi-jump causal and association reasoning on the dynamic knowledge graph, identifying the root cause entity, the influence diffusion path and the key association factors, and generating an alternative decision set from a strategy library in a matching way based on the reasoning result.
  2. 2. The intelligent analysis early warning and decision support method based on big data processing according to claim 1 is characterized by further comprising a step 6 of performing utility prediction on each decision, converting the decision with optimal utility into an executable instruction and sending the executable instruction to an execution terminal, and continuously collecting an effect index after decision execution as a reinforcement learning reward signal for optimizing the model weight in the step 4, the strategy generation model in the step 5 and the fusion weight in the step 2.
  3. 3. The intelligent analysis early warning and decision support method based on big data processing according to claim 2 is characterized in that in step 1, the real-time stream data is accessed and analyzed in millisecond level by APACHEFLINK or SparkStreaming; the preset format is entity ID, time stamp, feature dimension 1, feature dimension 2, feature dimension N and data source label.
  4. 4. The intelligent analysis early warning and decision support method based on big data processing according to claim 3, wherein the time-varying weight adaptive fusion algorithm in the step 2 is specifically: Unifying feature vectors for the feature of entity e at time t From real-time feature vectors And historical feature vector Linear fusion, the formula is as follows: ; wherein, the weight is fused Is a time-varying function, and the calculation formula is as follows: ; Wherein the method comprises the steps of For the sigmoid activation function, Is the sensitivity coefficient of the sensor, Is entity e in time window Instantaneous rate of change in; is the rate of change threshold.
  5. 5. The intelligent analysis early warning and decision support method based on big data processing according to claim 4, wherein the dynamic knowledge graph updating in the step 3 adopts a mode of combining event driving and period mining; the dynamic knowledge graph takes a business entity as a node, takes a multi-element relation among the entities as an edge, and adds a graph structure of weight and time stamp; triggering map edge weight update or new edge and node addition when an associated event is generated in the real-time stream, mining hidden relations in the historical data by adopting a map mining algorithm every other preset period, supplementing the hidden relations into a knowledge map, and perfecting a map structure; And selecting 1-3 hop neighbor nodes by taking a target entity as a center, carrying out weighted aggregation on the characteristics of the target entity, the characteristics of the neighbor nodes and the edge weight information through a graph neural network, outputting an enhanced characteristic vector with the dimension consistent with the unified characteristic vector, and retaining the associated context information among the entities.
  6. 6. The intelligent analysis early warning and decision support method based on big data processing according to claim 5, wherein the weighted consensus decision in step 4 is specifically: m early warning analysis models are arranged, for an entity e at time t, the early warning result output by the ith model is (flag, score), and the generation rule of the comprehensive early warning signal Alert is as follows: ; where M is the number of pre-alarm analysis models, Is the early warning mark of the ith model, and takes the value as Wherein 1 represents that the model outputs an early warning signal, and 0 represents that no early warning exists; is the confidence level of the first model output warning, Is the dynamic weight of the first model at time t, Is a global early warning value.
  7. 7. The intelligent analysis early warning and decision support method based on big data processing according to claim 6, wherein the multi-hop causal and association reasoning in the step 5 adopts a random walk and influence propagation model based on a meta-path, the decision utility prediction is calculated by constructing a multi-objective utility function, the simulation deduction adopts a simulation environment based on an intelligent agent, and the meta-path is a predefined entity relationship path template and is used for restraining the random walk direction and focusing the association path related to the service.
  8. 8. The intelligent analysis early warning and decision support method based on big data processing of claim 7, wherein random walk based on a meta-path takes an early warning entity as a starting point, multi-jump walk is carried out in a knowledge graph according to a meta-path template, occurrence probability of each path is counted, a core associated entity and a causal path are identified, an influence propagation model is based on an independent cascade model, influence intensity of the early warning entity on the associated entity is calculated, and the formula is: Wherein As the probability of influence of entity u on v, The early warning intensity of the early warning entity u; cost, coverage rate, expected benefit and risk attenuation factor construction of multi-objective utility function comprehensive decision, wherein the formula is as follows: wherein R is the expected benefit, C is the cost of execution, For coverage, R is a risk value, The sum is 1, which is the weight coefficient.
  9. 9. The intelligent analysis early warning and decision support method based on big data processing according to claim 8, wherein reinforcement learning optimization in the step 6 adopts an Actor-Critic framework, and the reward signal r (t) is formed by normalizing the difference between the key performance index variation after decision execution and the execution cost.
  10. 10. An intelligent analysis early warning and decision support system based on big data processing is used for realizing the method of claim 9, and is characterized by comprising a data lake layer, a fusion calculation layer, a knowledge map layer, an intelligent engine layer and an application feedback layer; the fusion calculation layer is connected with the data lake bin layer and is used for executing flow batch fusion calculation and unified feature vector generation; the knowledge graph layer is connected with the fusion calculation layer and the data lake storehouse layer and is used for constructing, storing and calculating a dynamic knowledge graph; The intelligent engine layer is connected with the fusion calculation layer and the knowledge map layer and is used for executing early warning analysis, decision reasoning and strategy optimization; the application feedback layer is connected with the intelligent engine layer and the execution terminal and is used for displaying early warning and decision information and collecting feedback data.

Description

Intelligent analysis early warning and decision support system and method based on big data processing Technical Field The invention relates to the technical field of big data intelligent analysis, in particular to an intelligent analysis early warning and decision support system and method based on big data processing. Background Along with the rapid development of big data technology, the intelligent analysis early warning and decision support system is widely applied to a plurality of fields such as financial wind control, supply chain management, industrial equipment predictive maintenance, smart city operation and the like. The core requirement of the system is to realize accurate analysis, early warning in time and scientific decision based on multi-source data, and provide reliable support for business operation. In the prior art, the intelligent analysis early warning and decision support system has a plurality of limitations of data processing and analysis decision. On the data processing level, the offline batch processing and the real-time stream processing are mostly in separate structures, and the offline batch processing and the real-time stream processing are respectively and independently operated, so that real-time decisions can only depend on instant data and lack of historical deep support, historical data analysis cannot incorporate the latest business situation, decision basis is difficult to unify, and the integrity and accuracy of analysis results are affected. In the early warning mechanism level, the early warning rules of the existing system are mostly set based on static threshold values, so that the problems of high false alarm rate, missed report key risk and the like are easy to occur, and the flexibility and the accuracy of early warning are insufficient. In the decision capability level, the analysis early warning and decision module operates in a linear pipeline, the decision proposal only depends on a simple condition judgment rule, lacks the deep reasoning capability of complex association relation and event cause and effect links between service entities, has limited intelligent level, and is difficult to cope with the decision requirement in complex service scenes. In addition, the existing system has the common problem of feedback closed loop deficiency, and the effect index after decision execution cannot be effectively used for back feeding system optimization, so that model parameters, early warning rules and decision logic are difficult to update iteratively according to actual operation effects, the adaptability and reliability of the system after long-term operation are gradually reduced, and the advanced requirements of business sustainable development on an intelligent decision system cannot be met. Disclosure of Invention In order to solve the technical problems, the invention provides an intelligent analysis early warning and decision support system and method based on big data processing. The following technical scheme is adopted: the intelligent analysis early warning and decision support method based on big data processing comprises the following steps: Step 1, analyzing, cleaning and converting accessed real-time stream data and batch historical data to unify time sequence feature vectors and attribute tags in a preset format; Step 2, performing windowed real-time calculation on the real-time stream data, and performing periodic depth aggregation calculation on batch historical data; step 3, constructing and updating a dynamic knowledge graph with business entities as nodes and the multiple relations among the entities as edges in real time, and carrying out context information enhancement by combining the neighborhood structure of the entities in the knowledge graph and the states of the related entities to form an enhanced feature vector; Step 4, the enhanced feature vector is input into a plurality of parallel early warning analysis models, wherein the early warning analysis models comprise an anomaly detection model, a trend prediction model and a business rule model; And 5, when the confidence level of the comprehensive early warning signal exceeds a preset threshold, taking the early warning entity as a starting point, carrying out multi-jump causal and association reasoning on the dynamic knowledge graph, identifying the root cause entity, the influence diffusion path and the key association factors, and generating an alternative decision set from a strategy library in a matching way based on the reasoning result. Optionally, the method further comprises a step 6 of performing utility prediction on each decision, converting the decision with optimal utility into an executable instruction and transmitting the executable instruction to an execution terminal, and continuously collecting an effect index after decision execution as a reinforcement learning reward signal for optimizing the model weight in the step 4, the strategy generation model in the s