Search

CN-122027535-A - Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method and device

CN122027535ACN 122027535 ACN122027535 ACN 122027535ACN-122027535-A

Abstract

The application provides a Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method, which comprises the steps of extracting monitor information comprising a unique identification code, a group number and a theme through a scan operation state @ KAFKALISTENER container, identifying removed monitors with a historical state for checking as candidate revocation groups, changing the states of the monitors in the candidate revocation groups from activation to freezing, recording freezing time and cleaning triggering conditions, executing cleaning and updating the cleaning conditions to cleaning conditions when the conditions are met, automatically activating the cleaned monitors when the cleaned monitors reappear and the recovery conditions are met, and simultaneously adapting different platform Kafka cleaning operations through a unified interface to realize unified treatment of the cross-platform consumption groups. The method can realize the automatic state management of Kafka consumers, support reversible operation and cross-platform compatibility, effectively reduce operation and maintenance cost and improve system reliability.

Inventors

  • XUE GANG

Assignees

  • 北京米可世界科技有限公司

Dates

Publication Date
20260512
Application Date
20260318

Claims (11)

  1. 1. A Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method is characterized by comprising the following steps: scanning an operating state @ KAFKALISTENER container, and extracting monitor information, wherein the monitor information comprises a unique monitor identification code, a monitor group number and a monitor theme; checking accounts according to the extracted listener information and the history state, and identifying the removed listener as a candidate revocation group; Changing the states of all monitors in the candidate revocation group from an activated state to a frozen state, and recording freezing time and cleaning triggering conditions; When the cleaning triggering condition is met, cleaning operation is executed, and the states of all monitors in the candidate revocation group are changed from a frozen state to a cleaning state; when the cleaned monitor reappears and the recovery condition is met, automatically changing the state of the cleaned monitor into an activated state; And the unified treatment of the cross-platform consumption group is realized by adapting the Kafka cleaning operation of different platforms through a unified interface.
  2. 2. The method according to claim 1, wherein the scanning the @ KAFKALISTENER container in a running state extracts listener information, specifically including: extracting a listener group number, a listener theme and a listener unique identification code from a listener registry; And generating a listener fingerprint for subsequent reconciliation and state recovery based on the extracted listener group number, the listener theme and the unique listener identification code.
  3. 3. The method of claim 1, wherein the identifying the removed listener as a candidate revocation group based on the extracted listener information reconciles with the historical state, further comprising: Calculating the difference between the snapshot of the current monitor and the snapshot of the history state, and determining a newly added and removed monitor group; The removed listener group is marked as a candidate revocation group.
  4. 4. The method according to claim 1, wherein changing the state of the candidate revocation group from the active state to the frozen state, and recording the freezing time and the cleaning trigger condition, comprises: Setting a freezing window, and observing the state of the candidate revocation group in the freezing window; recording freezing time, and setting cleaning trigger time as a trigger point of hard cleaning; the cleaning trigger condition includes reaching a cleaning trigger time, being in a low peak time window, not exceeding a single round of cleaning threshold, and passing an active member access check.
  5. 5. The method according to claim 4, wherein the performing a cleaning operation when the cleaning trigger condition is satisfied, comprises: checking whether the current time reaches the cleaning trigger time; checking whether the current time is in a preset low peak time window; checking whether the cleaning quantity of the current round exceeds a preset single round cleaning threshold value; if the consumption group member is in the manager mode, inquiring whether the consumption group member has an active member; The cleaning operation is performed only when all cleaning trigger conditions are satisfied at the same time.
  6. 6. The method according to claim 1, wherein when the cleaned listener reappears and the recovery condition is met, automatically changing the state of the cleaned listener to an active state, in particular comprising: Monitoring the state change of the cleaned monitor; When detecting that the cleaned monitor reappears, verifying whether the cleaned monitor meets a recovery condition or not, wherein the recovery condition comprises token matching; If the recovery condition is satisfied, the state is automatically changed from the cleared state to the activated state.
  7. 7. The method according to claim 1, wherein the adapting Kafka cleaning operations of different platforms through a unified interface realizes unified governance of cross-platform consumer groups, specifically comprising: Defining a unified cleaning interface specification, and isolating the implementation differences of different platforms Kafka; Cleaning operation adapters for realizing self-built Kafka and cloud platform Kafka; and by configuring and switching different adapters, unified treatment of the cross-platform consumption group is realized.
  8. 8. The method as recited in claim 1, further comprising: providing a state query interface, allowing a user to query the current state and historical state change records of the consumption group; A failure cause tracking interface is provided to allow a user to query for a specific cause of failure of the cleaning operation.
  9. 9. The method as recited in claim 1, further comprising: configuring treatment parameters, wherein the treatment parameters comprise a freezing window, a low peak time window, a cleaning speed limit and a rollback retry interval; And dynamically adjusting the treatment parameters according to the actual running conditions so as to optimize the treatment effect.
  10. 10. The method according to claim 1, wherein a staged online strategy is employed in the deployment of the production environment, comprising: the primary verification stage adopts a file storage mode to verify and treat the closed loop; switching to a database storage mode in the formal production stage to ensure the consistency of states among multiple instances; for a self-built Kafka cluster, an active member check function of an administrator mode is enabled.
  11. 11. Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning device, which is characterized by comprising: the monitor scanning module is used for scanning the @ KAFKALISTENER container in the running state and extracting monitor information, wherein the monitor information comprises a unique monitor identification code, a monitor group number and a monitor theme; The account checking identification module is connected with the monitor scanning module and is used for checking accounts according to the extracted monitor information and the historical state, and the removed monitor is identified as a candidate revocation group; The state freezing module is connected with the account checking identification module and is used for changing the states of all monitors in the candidate revocation group from an activated state to a frozen state, and recording freezing time and cleaning triggering conditions; the cleaning execution module is connected with the state freezing module and is used for executing cleaning operation when the cleaning triggering condition is met, and changing the states of all monitors in the candidate revocation group from a freezing state to a cleaning state; The state recovery module is connected with the cleaning execution module and is used for automatically changing the state of the cleaned monitor into an activated state when the cleaned monitor reappears and the recovery condition is met; And the platform adaptation module is respectively connected with the cleaning execution module and the state recovery module and is used for adapting the Kafka cleaning operation of different platforms through a unified interface to realize the unified treatment of the cross-platform consumption group.

Description

Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method and device Technical Field The invention relates to the technical field of Kafka message middleware consumer management for Spring frames, in particular to a Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method and device. Background In the field of distributed message systems, integration of a Spring framework and Kafka has become a mainstream technical scheme for implementing asynchronous communication in a micro-service architecture. The Spring-based @ KAFKALISTENER annotation mechanism can quickly construct message consumers and realize high availability of load balancing and message processing through consumption group management. In current technical practice, the operation and maintenance management of Kafka consumers typically relies on manual intervention or platform-specific automation scripts, such as scanning consumer group status through timed tasks, manually triggering offline operations, or developing cleaning tools for different Cloud platforms (e.g., conflux Cloud, AWS MSK, etc.) respectively. However, the prior art solutions have significant limitations. On one hand, the consumer state change lacks a reversible control mechanism, when the consumer state is misplaced or the business requirement is changed, the cleaned consumer group cannot be automatically recovered, the application needs to be redeployed and the consumer group needs to be rebuilt, so that the service interruption risk and the operation and maintenance cost are increased, on the other hand, the API difference and the authority model difference of different Kafka service providers in a cross-platform scene make the consumer group cleaning operation difficult to standardize, and enterprises need to maintain independent management tool chains for each platform, so that the technical complexity is increased, and the flexible deployment capability of the multi-cloud architecture is limited. In addition, the traditional account checking mode relies on static configuration comparison, so that the actual state of a running state consumer is difficult to be perceived dynamically, and the problem of repeated consumption or loss of messages caused by inconsistent states is easily caused. Therefore, a unified governance scheme capable of realizing refined and reversible management of consumer states and shielding platform differences is needed. The scheme needs to have dynamic state sensing, automatic freezing-cleaning-recovery flow control and cross-platform operation adaptation capability, so that the operation and maintenance complexity of the distributed system is reduced while the reliability of message processing is ensured, and an enterprise is supported to construct a high-elasticity and low-cost micro-service architecture. Disclosure of Invention In order to solve the existing technical problems, the application provides a Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method and device. In a first aspect, an embodiment of the present application provides a Spring-oriented Kafka consumer reversible automatic offline and cross-platform cleaning method, including: Scanning an operating state @ KAFKALISTENER container, and extracting monitor information, wherein the monitor information comprises a unique identifier of a monitor, a monitor group number and a monitor theme; Checking accounts according to the extracted listener information and the history state, and identifying the removed listener as a candidate revocation group; Changing the states of all monitors in the candidate revocation group from an activated state to a frozen state, and recording freezing time and cleaning triggering conditions; when the cleaning trigger condition is met, cleaning operation is executed, and the states of all monitors in the candidate revocation group are changed from the frozen state to the cleaning state; When the cleaned monitor reappears and the recovery condition is met, automatically changing the state of the cleaned monitor into an activated state; And the unified treatment of the cross-platform consumption group is realized by adapting the Kafka cleaning operation of different platforms through a unified interface. Optionally, the @ KAFKALISTENER container in a running state is scanned, and the listener information is extracted, which specifically includes: extracting a listener group number, a listener theme and a listener unique identification code from a listener registry; And generating a listener fingerprint for subsequent reconciliation and state recovery based on the extracted listener group number, the listener topic, and the listener unique identification code. Optionally, reconciling the extracted listener information with the history state, identifying the removed listener as a candidate revocation group, and further comprising: Calculating the dif