Search

CN-121979700-A - Method and system for preventing loop-back bidirectional message synchronization among message queue clusters

CN121979700ACN 121979700 ACN121979700 ACN 121979700ACN-121979700-A

Abstract

The invention relates to the technical field of distributed data processing, and discloses a method and a system for preventing loop-back bidirectional message synchronization among message queue clusters, wherein the method comprises the steps that a message producer client determines the current cluster identity according to network connection configuration, and the message producer client sets a source cluster identity attribute in a message object; the synchronization component pulls the information object from the first message queue cluster, the synchronization component adds a synchronization source mark attribute before forwarding to the second message queue cluster, the synchronization component runs a reverse synchronization task, discards the message object if the message object is detected to contain the synchronization source mark attribute, and the message consumer client compares the source cluster identification attribute with the local cluster identification, and sends a consumption confirmation and discards the message object if the source cluster identification attribute is inconsistent with the local cluster identification. The invention realizes self-contained routing judgment through the message object metadata without depending on an external global state database, reduces the complexity of the architecture and improves the throughput performance of the synchronous link.

Inventors

  • ZHOU SHUCHENG

Assignees

  • 人保信息科技有限公司

Dates

Publication Date
20260505
Application Date
20251217

Claims (10)

  1. 1. A method for preventing loop-back bidirectional message synchronization among message queue clusters is characterized by comprising the following steps: S100, a message producer client sends a message object to a first message queue cluster, the message producer client sets a source cluster identification attribute in the message object through a customized software development kit, an attribute value of the source cluster identification attribute is set as a unique identifier of the first message queue cluster, and the message object is stored in a message subject of the first message queue cluster; S200, a synchronization component starts a message synchronization task from the first message queue cluster to a second message queue cluster, the synchronization component pulls the message object from the first message queue cluster, the synchronization component executes an attribute enhancement operation on the message object before forwarding the message object to the second message queue cluster, the attribute enhancement operation comprises reading an attribute value of the source cluster identification attribute in the message object, adding a synchronization source mark attribute in the message object, setting the attribute value of the synchronization source mark attribute to be the same as the attribute value of the source cluster identification attribute, and then writing the message object into the second message queue cluster; And S300, the synchronization component runs a reverse synchronization task from the second message queue cluster to the first message queue cluster, the reverse synchronization task pulls the message object from the second message queue cluster and executes loop detection logic, and if the message object already contains the synchronization source mark attribute, the reverse synchronization task suspends synchronization operation and discards the message object.
  2. 2. The method for preventing loop-back bidirectional message synchronization between message queue clusters according to claim 1, wherein in S100, the custom software development kit reads client configuration information in an initialization stage of the message producer client, the custom software development kit extracts name server address parameters from the client configuration information, the name server address parameters characterize a target service endpoint of the message producer client currently establishing a transmission control protocol connection, the custom software development kit loads a preset cluster address mapping table, and the custom software development kit performs matching calculation on the name server address parameters and the cluster address mapping table, thereby outputting a current cluster identity.
  3. 3. The method for bidirectional message synchronization between message queue clusters according to claim 2, wherein in S100, the customized software development kit responds to a message sending instruction of the message producer client, the customized software development kit generates an instance of the message object, and the customized software development kit registers a source cluster identification field in a user-defined attribute dictionary included in the message object; the customized software development kit acquires the current cluster identity, and writes the current cluster identity into an entry with a key name SEND_AZ in the user-defined attribute dictionary.
  4. 4. The method of claim 1, wherein in S200, the synchronization component initializes a network communication session with the first message queue cluster, the synchronization component registers a synchronization-specific consumer group in the first message queue cluster, the synchronization-specific consumer group subscribes to a message topic to be synchronized, and the synchronization component initiates a batch pull request to the first message queue cluster according to a locally recorded consumption site; the synchronization component receives a message object list, the synchronization component stores the message object list in an internal memory buffer of the synchronization component, and the synchronization component updates the locally recorded value of the consumption site after confirming that the message object list is successfully received.
  5. 5. The method of claim 1, wherein in S200, the synchronization component performs detection decision logic after pulling the message object and before performing the attribute enhancement operation: The synchronization component reads a user attribute set in the message object, traverses the user attribute set to acquire all attribute key names, and detects whether the attribute key names contain predefined synchronization source mark attribute key names or not; when the synchronization component detects that the user attribute set contains the synchronization source mark attribute key name, the synchronization component judges that the message object is a loop-back message, and the synchronization component destroys the message object in a memory; When the synchronization component detects that the user attribute set does not contain the synchronization source mark attribute key name, the synchronization component judges the message object as a native message, and the synchronization component allows the message object to enter the attribute enhancement operation.
  6. 6. The method for bidirectional message synchronization between message queue clusters according to claim 1, wherein in S200, the synchronization component connects to the second message queue cluster using a pre-established network channel, the synchronization component encapsulates the message object subjected to the attribute enhancement operation as a data transmission request, and the synchronization component invokes a message transmission interface to transmit the data transmission request to the second message queue cluster; And the synchronization component receives an execution result, updates the consumption progress cursor of the first message queue cluster when the execution result indicates that the writing is successful, and executes retry sending logic for the message object when the execution result indicates that the writing is failed.
  7. 7. The method for anti-loop-back two-way message synchronization between message queue clusters according to claim 1, further comprising a local environment decision performed by a message consumer client, comprising the steps of: The message consumer client reads configuration information of the current active network connection, the message consumer client extracts a target service end access point address from the configuration information, the message consumer client performs matching operation on the target service end access point address and a preset cluster address list, and the message consumer client determines a local cluster identifier according to a matching result.
  8. 8. The method for anti-loop-back bi-directional message synchronization between message queue clusters of claim 7, further comprising a homologous consumption policy performed by a message consumer client, comprising the steps of: The message consumer client subscribes to and pulls the message object from the first message queue cluster or the second message queue cluster, reads an attribute value with a key name SEND_AZ in the message object, compares the attribute value with the local cluster identifier, triggers business logic to consume the message object if the attribute value is consistent with the local cluster identifier, and discards the message object locally if the attribute value is inconsistent with the local cluster identifier.
  9. 9. The method for preventing loop-back bi-directional message synchronization between message queue clusters according to claim 8, wherein said message consumer client locally discards said message object, comprising the steps of: The message consumer client sends a consumption confirmation data packet to a message proxy server node of the first message queue cluster or the second message queue cluster, wherein the consumption confirmation data packet informs the message proxy server node that the message object is successfully processed, but the message consumer client does not call a business layer processing function.
  10. 10. A loop-back prevention bidirectional message synchronization system among message queue clusters, which is characterized by being applied to the loop-back prevention bidirectional message synchronization method among the message queue clusters according to any one of claims 1-9, and comprising a first message queue cluster, a second message queue cluster, a message producer client, a message consumer client and a synchronization component; the message producer client is integrated with a message production module, and the message production module is used for reading network connection configuration so as to determine a physical cluster identity of the current client, and writing the identity into an attribute item with a key name SEND_AZ in a user attribute set when generating the message object; The synchronization component is integrated with a message synchronization module, and the message synchronization module is used for pulling the message object from a source message queue cluster, checking whether a synchronization source mark attribute with a key name of ORIGIN_SEND_UNIT exists in the user attribute set, discarding if the synchronization source mark attribute exists, extracting an attribute value with the key name of SEND_AZ in the message object if the synchronization source mark attribute does not exist, constructing an attribute with the key name of ORIGIN_SEND_UNIT, assigning the extracted attribute value to the attribute with the key name of ORIGIN_SEND_UNIT, and sending the processed message object to a destination message queue cluster; The message consumer client is integrated with a message consumption module, and the message consumption module is used for extracting an attribute value with a key name SEND_AZ in the message object, acquiring the identity of the local cluster, and intercepting and discarding the message object locally if the attribute value is inconsistent with the identity of the local cluster.

Description

Method and system for preventing loop-back bidirectional message synchronization among message queue clusters Technical Field The invention relates to the technical field of distributed data processing, in particular to a method and a system for preventing loop-back bidirectional message synchronization among message queue clusters. Background In a large-scale distributed system, in order to ensure high availability and disaster recovery capability of data, a plurality of independent message queue clusters are required to be deployed in physical resource pools of different geographic positions, and bidirectional message synchronization is required between a first message queue cluster in a first physical resource pool and a second message queue cluster in a second physical resource pool, so that service data are ensured to be consistent among different regions. When a bidirectional synchronous channel is established between the first message queue cluster and the second message queue cluster, the message objects which are synchronized from the first message queue cluster to the second message queue cluster are easily acquired again by a synchronization mechanism of the second message queue cluster and are returned to the first message queue cluster, so that message routing endless loop is formed. The existing loop-back prevention technology relies on maintaining an external global state database in a synchronous link to record a message transmission track or relies on a complex distributed lock mechanism to coordinate a synchronous state, the external global state database introduces additional architecture complexity and maintenance cost, and in a high concurrency scene, frequent access to the external global state database increases network delay, so that the throughput performance of the message synchronous link is limited. The existing message production and transmission mechanism lacks the capability of solidifying physical attribution information at the source of message object generation, the message object lacks self-contained initial generation evidence in the process of cross-network transmission or storage, the subsequent processing links are difficult to judge the original source of the message object according to the data content of the message object, and when the network topology is changed or the configuration is drifted, the message object lacking the solidified evidence can cause the failure of the loop-back prevention judgment logic. In the scenario of realizing full data backup, the first message queue cluster and the second message queue cluster both contain full message data through synchronization, if the message consumer client cannot distinguish whether the message object is a native message produced locally or a copy message synchronized from different places, the message consumer client can execute service logic processing on all the message objects, repeated service operation on the message objects synchronized from different places can cause resource waste, so that service state is abnormal, and the final consistency of the service data between the first message queue cluster and the second message queue cluster is further destroyed. Disclosure of Invention Aiming at the defects of the prior art, the invention provides a method and a system for preventing loop-back bidirectional message synchronization among message queue clusters, which solve the problems that the prior art depends on an external global state database to cause complex architecture, the throughput performance of a synchronous link is limited, the original source is difficult to judge due to lack of a solidified original generation credential of a message object, and a message consumer client cannot distinguish different synchronous messages to cause repeated consumption so as to destroy data consistency. To achieve the above object, a first aspect of the present invention provides a method for preventing loop-back bidirectional message synchronization between message queue clusters, comprising the steps of: S100, in the generation stage, a message producer client sends a message object to a first message queue cluster, the message producer client sets a source cluster identification attribute in the message object through a customized software development kit, an attribute value of the source cluster identification attribute is set as a unique identifier of the first message queue cluster, and the message object is stored in a message subject of the first message queue cluster; S200, in a synchronization stage, a synchronization component starts a message synchronization task from a first message queue cluster to a second message queue cluster, the synchronization component pulls a cancel object from the first message queue cluster, the synchronization component executes an attribute enhancement operation on the message object before forwarding the message object to the second message queue cluster, the attribut