CN-122001943-A - Data real-time pushing method based on producer and consumer association

CN122001943ACN 122001943 ACN122001943 ACN 122001943ACN-122001943-A

Abstract

The invention relates to the technical field of data communication, and discloses a method and a system for pushing data in real time based on association of a producer and a consumer. The method comprises the steps of constructing a system comprising an association registration module, a data acquisition module, an association matching module, a push scheduling module, a fault-tolerant processing module and a monitoring operation and maintenance module, wherein a producer and a consumer register and submit attribute, demand and terminal information respectively, the association matching module establishes a dynamic association model through semantic analysis and feature extraction to generate an association relation table, the data acquisition module acquires and preprocesses producer data in real time, the push scheduling module pushes the data to a target terminal in real time according to the association relation table and a data attribute tag by adopting a distributed scheduling strategy, the fault-tolerant processing module monitors and processes push abnormality, and the monitoring operation and maintenance module monitors all links and generates reports. The invention realizes dynamic and accurate association matching and data pushing, and remarkably improves pushing instantaneity, reliability and system resource utilization rate.

Inventors

WEI YONG
LI YANFENG
Jia Xiaopu
CUI JUNFENG
Hao Ruize

Assignees

中国人民解放军63729部队

Dates

Publication Date: 20260508
Application Date: 20260128

Claims (8)

1. The real-time pushing method for the data based on the association of the producer and the consumer is characterized by comprising the following steps: S1, constructing a producer-consumer association management system, wherein the system comprises an association registration module (10), a data acquisition module (20), an association matching module (30), a push scheduling module (40), a fault tolerance processing module (50) and a monitoring operation and maintenance module (60); S2, registering by the producer through the association registration module (10), and submitting attribute information of the producer; the consumer registers through the association registration module (10), submits consumer demand information and receiving terminal information; S3, the association matching module (30) performs semantic analysis and feature extraction on the producer attribute information and the consumer demand information, establishes a dynamic association model and generates a producer-consumer association relation table; S4, the data acquisition module (20) acquires data output by a producer in real time, and marks a data attribute label after preprocessing the data; S5, the pushing scheduling module (40) determines a target consumer according to the association relation table and the data attribute label, adopts a distributed scheduling strategy to allocate pushing resources, and pushes the preprocessed data to a receiving terminal of the target consumer in real time; s6, the fault-tolerant processing module (50) monitors the pushing process in real time, and when pushing abnormality occurs, a corresponding fault-tolerant mechanism is triggered; And S7, the monitoring operation and maintenance module (60) monitors the system running state, the change of the producer-consumer association relation and the data pushing performance in real time, and generates an operation and maintenance report.
2. The method according to claim 1, wherein in step S2, the producer attribute information includes producer ID, data type, data output frequency, data format, data source address and authentication information, the consumer demand information includes consumer ID, required data type, data update frequency requirement, data precision requirement and screening condition, and the receiving terminal information includes terminal IP address, port number, terminal type and communication protocol.
3. The method for pushing data based on producer and consumer association in real time according to claim 1, wherein in step S3, the specific process of establishing the dynamic association model by the association matching module (30) comprises the following steps: S31, preprocessing the attribute information of the producer and the demand information of the consumer, and removing redundant information and noise data; S32, carrying out semantic analysis on the preprocessed information by adopting a semantic model, and extracting core data characteristics of a producer and core demand characteristics of a consumer; s33, calculating the matching degree of the core data characteristic and the core demand characteristic based on the two characteristics; S34, setting a matching degree threshold, and when the calculated matching degree is greater than or equal to the threshold, establishing an association relation between a producer and a consumer and inputting association information into the association relation table; And S35, monitoring the change of the attribute information of the producer and the demand information of the consumer in real time, and when the information is changed, recalculating the matching degree and updating the association relation table.
4. The method for pushing data based on the association of the producer and the consumer according to claim 1, wherein in the step S4, the data preprocessing comprises data cleaning, data standardization and data compression, the data cleaning is used for removing abnormal values, repeated values and missing values in the data, the data standardization is used for converting the data in different formats into a uniform format, and the data compression is carried out by adopting a compression algorithm.
5. The method for pushing data in real time based on the association of a producer and a consumer according to claim 1, wherein in step S5, the distributed scheduling policy specifically comprises: s51, dividing the system into a plurality of pushing nodes, wherein each pushing node is responsible for a pushing task of a certain area or a certain type of consumer; S52, the push scheduling module (40) collects the load state of each push node in real time, wherein the load state comprises CPU utilization rate, memory occupancy rate, network bandwidth utilization rate and the current number of push tasks; S53, distributing pushing tasks by adopting an optimization algorithm according to the load state and the association relation table so as to balance the load of each pushing node; and S54, the pushing node establishes long connection with the consumer receiving terminal by adopting a WebSocket protocol according to the distributed pushing task, or pushes the preprocessed data to the terminal in real time by adopting an HTTP long polling mode.
6. The method for pushing data based on the association of the producer and the consumer according to claim 1, wherein in step S6, the fault tolerant mechanism comprises a data retransmission mechanism, a node redundancy backup mechanism and a failover mechanism; when the data retransmission mechanism detects data pushing failure, if the failure times are smaller than a preset threshold value, adopting an exponential backoff algorithm to retransmit the data, and if the failure times are larger than or equal to the preset threshold value, handing over the pushing task to a standby pushing node; the node redundancy backup mechanism configures at least one standby node for each pushing node, and the standby nodes synchronize the pushing tasks and the associated information of the main node in real time; The failover mechanism triggers failover when the fault-tolerant processing module (50) detects a failure of the primary push node, and the backup node takes over the push task of the primary node.
7. A producer and consumer association based data real-time push system based on the method of any of claims 1-6, comprising: the associated registration module (10) is used for receiving registration information of a producer and a consumer and verifying and storing the registration information; the data acquisition module (20) is used for acquiring data output by a producer in real time, and preprocessing the data and marking attribute labels; The association matching module (30) is used for carrying out semantic analysis and feature extraction on the attribute information of the producer and the demand information of the consumer, establishing a dynamic association model and generating and maintaining an association relation table; the pushing scheduling module (40) is used for determining a target consumer according to the association relation table and the data attribute label, distributing pushing resources by adopting a distributed scheduling strategy and realizing real-time pushing of data; the fault-tolerant processing module (50) is used for monitoring the pushing process in real time, and triggering a corresponding fault-tolerant mechanism when pushing abnormality occurs; And the monitoring operation and maintenance module (60) is used for monitoring the system running state, the change of the producer-consumer association relation and the data pushing performance in real time and generating an operation and maintenance report.
8. The system according to claim 7, wherein the associated registration module (10) further comprises an identity authentication unit (11) for authenticating registration information of the producer and the consumer, and the data acquisition module (20) further comprises a data quality detection unit for performing quality detection on the preprocessed data, wherein the indexes of the quality detection include data integrity, accuracy and timeliness.

Description

Data real-time pushing method based on producer and consumer association Technical Field The invention relates to the technical field of data communication, in particular to a data real-time pushing method based on the association of a producer and a consumer. Background In the digital age, data is taken as a core production element, and the circulation efficiency directly influences the service response speed and the user experience. The data pushing is used as a key link of data circulation and is widely applied to various fields such as instant messaging, real-time monitoring, online service and the like. The existing data pushing method is mostly realized based on broadcasting or point-to-point mode, wherein broadcasting is used for sending data to all potential receivers indifferently, a large amount of invalid data transmission exists, excessive network bandwidth and terminal resources are occupied, the point-to-point pushing can be used for realizing accurate sending, one-to-one communication connection is required to be established, when the number of producers and consumers is huge, the complexity of connection management is exponentially increased, and the change of the demands of the consumers and the fluctuation of the data output of the producers are difficult to flexibly cope with. With the development of technologies such as the internet of things and cloud computing, the number of data producers and consumers is increased, and the association relationship between the data producers and consumers is characterized by dynamic change. For example, in an intelligent home scene, environmental data generated by a sensor (producer) needs to be pushed to a corresponding control terminal (consumer), the number and the type of the terminal can change at any time, and in an electronic commerce platform, commodity information update of a merchant (producer) needs to be pushed to a user (consumer) focusing on the merchant, and the focusing relationship of the user can be adjusted in real time. The prior art is difficult to efficiently process the real-time data push requirement under the dynamic association, and has the problems of high push delay, low resource utilization rate, chaotic association relation management and the like. To solve the above-mentioned problems, some improvements based on subscription-publishing modes are proposed in the related art. For example, by setting up a message broker node, the producer publishes data to the message broker from which the consumer subscribes to the data. However, in such a scheme, the message broker usually adopts a centralized architecture, which is easy to become a performance bottleneck, and when the association relationship between the producer and the consumer is complex, the configuration and maintenance of the subscription rule are more difficult. In addition, part of schemes introduce a distributed architecture, but the aspects of data slicing, node coordination, dynamic load balancing and the like still have the defects, and the real-time performance and the reliability of data pushing are difficult to ensure. In addition, in the prior art, when the association relation between the producer and the consumer is processed, static configuration or a simple keyword matching mode is mostly adopted, and deep association between the producer and the consumer cannot be accurately identified, so that the accuracy of the pushed data is lower. Meanwhile, when the network environment fluctuates or the nodes are in fault, the existing scheme lacks an effective fault-tolerant mechanism, so that data loss or push interruption is easy to cause, and the continuity of the service is influenced. Therefore, how to construct a data pushing method capable of dynamically managing the association relation between a producer and a consumer and giving consideration to real-time performance, reliability and resource utilization rate becomes a technical problem to be solved in the current field. Disclosure of Invention The invention aims to overcome the defects of the prior art, provides a data real-time pushing method based on the association of a producer and a consumer, the method can dynamically establish and maintain the association relation between the producer and the consumer, realize accurate and real-time pushing of data, and improve the resource utilization rate and the reliability of the system. In order to achieve the above purpose, the invention adopts the following technical scheme: a data real-time pushing method based on producer and consumer association comprises the following steps: S1, constructing a producer-consumer association management system, wherein the system comprises an association registration module, a data acquisition module, an association matching module, a push scheduling module, a fault tolerance processing module and a monitoring operation and maintenance module; S2, registering by a producer through an association registration module an