Search

CN-121984927-A - Log delivery system and method

CN121984927ACN 121984927 ACN121984927 ACN 121984927ACN-121984927-A

Abstract

The embodiment of the invention discloses a log delivery system and a method, which are used for acquiring logs from a source Kafka cluster by starting consumer threads, distributing the logs to corresponding buffer queues according to target cluster identifiers in the logs, and delivering the logs positioned at the heads of the buffer queues to corresponding user Kafka clusters through producer threads in thread pools corresponding to the buffer queues, wherein the capacity of the thread pools is dynamically adjusted according to the query number per second and/or the data backlog quantity of the buffer queues.

Inventors

  • LV ZHENYU

Assignees

  • 北京比特易湃信息技术有限公司

Dates

Publication Date
20260505
Application Date
20260204

Claims (10)

  1. 1. A log delivery system comprising a source Kafka cluster and at least one user Kafka cluster, the server on which the source Kafka cluster resides deploying a delivery service comprising: The message consumption module is used for starting a consumer thread and acquiring a log from the source Kafka cluster; the message distribution module is used for distributing the log to the corresponding message delivery module according to the target cluster identifier in the log; and the message delivery modules are used for storing the logs into a buffer queue so that producer threads in a thread pool can deliver the logs positioned at the head of the buffer queue to corresponding user Kafka clusters, wherein each message delivery module corresponds to different user Kafka clusters respectively, and the capacity of the thread pool is dynamically adjusted according to the query number per second and/or the data backlog quantity of the buffer queue.
  2. 2. The system of claim 1, wherein the delivery service is further configured to create a connection pool including a predetermined number of network connections corresponding to each of the user Kafka clusters, and wherein the producer thread delivers logs to the corresponding user Kafka cluster by multiplexing the network connections in the connection pool.
  3. 3. The system of claim 1, wherein the delivery service further comprises a blacklist management module for storing and maintaining a blacklist, wherein the blacklist includes a user Kafka cluster with a fault and/or a subject of the fault in a user Kafka cluster.
  4. 4. The system of claim 1, wherein the message distribution module is further configured to: Determining a corresponding user Kafka cluster according to the target cluster identification in the log; And determining a corresponding message delivery module according to the user Kafka cluster and a mapping relation table, wherein the mapping relation table is used for representing the corresponding relation between the user Kafka cluster and the message delivery module.
  5. 5. The system of claim 4, wherein the message delivery module comprises at least one thread pool and at least one buffer queue corresponding to the at least one thread pool, each of the thread pool and the buffer queue corresponding to a topic in the user Kafka cluster, the message delivery module further configured to: determining a target theme according to the target theme identification carried in the log; determining a corresponding thread pool and a buffer queue according to the target subject; Storing the log into a corresponding buffer queue; And the producer thread responds to the existence of the log in the corresponding buffer queue, and delivers the log positioned at the head of the buffer queue to the corresponding user Kafka cluster.
  6. 6. The system of claim 1, wherein the producer thread bulk posts logs in the buffer queue based on a maximum delay time, a batch size upper bound, and a compression algorithm.
  7. 7. The system of claim 1, wherein the delivery service further comprises: the monitoring module is used for sending index information and abnormal information in the log delivery process to the user side.
  8. 8. The system of claim 7, wherein the monitoring module is further configured to: and filtering the same type of abnormal information in a preset time in response to the captured abnormal information.
  9. 9. A log delivery method, the method comprising: Starting a consumer thread and acquiring a log from a source Kafka cluster; distributing the log to a corresponding buffer queue according to a target cluster identifier in the log; and delivering the log positioned at the head of the buffer queue to a corresponding user Kafka cluster through the producer threads in the thread pool corresponding to the buffer queue, wherein the capacity of the thread pool is dynamically adjusted according to the query number per second and/or the data backlog quantity of the buffer queue.
  10. 10. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of claim 9.

Description

Log delivery system and method Technical Field The present invention relates to the field of computer technology, and more particularly, to a log delivery system and method. Background The Kafka cluster is a distributed message flow processing platform constructed based on Kafka open source software and is used for realizing message publishing and subscribing. In the prior art, in order to achieve log delivery from a source Kafka cluster to each user Kafka cluster, a transfer Kafka cluster is generally used as a transfer message middleware, and meanwhile, in order to achieve data transmission and screening between two Kafka clusters, a link cluster needs to be set between the source Kafka cluster and the transfer Kafka cluster, and between the transfer Kafka cluster and the user Kafka cluster, so that log screening and delivery are achieved. The logs to be delivered in the architecture need to sequentially pass through links of 'source Kafka cluster-Flink cluster-transit Kafka cluster-Flink cluster-user Kafka cluster', and in the process of processing through the Flink cluster twice, serialization and anti-serialization operations are needed to be carried out on the logs, so that additional performance expenditure is generated, and the additional performance expenditure is combined with a longer transmission link, so that the log delivery delay is obviously increased, and the requirement of a low-delay service scene is difficult to meet. Disclosure of Invention In view of the above, the embodiments of the present invention provide a log-delivery system and method, so as to simplify a data transmission path and reduce a delay of a log-delivery process. In a first aspect, an embodiment of the present invention provides a log delivery system, where the log delivery system includes a source Kafka cluster and at least one user Kafka cluster, where a delivery service is deployed on a server where the source Kafka cluster is located, where the delivery service includes: The message consumption module is used for starting a consumer thread and acquiring a log from the source Kafka cluster; the message distribution module is used for distributing the log to the corresponding message delivery module according to the target cluster identifier in the log; and the message delivery modules are used for storing the logs into a buffer queue so that producer threads in a thread pool can deliver the logs positioned at the head of the buffer queue to corresponding user Kafka clusters, wherein each message delivery module corresponds to different user Kafka clusters respectively, and the capacity of the thread pool is dynamically adjusted according to the query number per second and/or the data backlog quantity of the buffer queue. Optionally, the delivery service is further configured to create a connection pool, where the connection pool includes a predetermined number of network connections corresponding to each of the user Kafka clusters, and the producer thread delivers logs to the corresponding user Kafka clusters by multiplexing the network connections in the connection pool. Optionally, the message distribution module is further configured to: And in response to the target cluster identification in the log being in a blacklist, destroying a thread pool and a buffer queue in a corresponding message delivery module, wherein the cluster in the blacklist is a user Kafka cluster with faults. Optionally, the message distribution module is further configured to: Determining a corresponding user Kafka cluster according to the target cluster identification in the log; And determining a corresponding message delivery module according to the user Kafka cluster and a mapping relation table, wherein the mapping relation table is used for representing the corresponding relation between the user Kafka cluster and the message delivery module. Optionally, the message delivery module includes at least one thread pool and at least one buffer queue corresponding to the at least one thread pool, each of the thread pool and the buffer queue corresponding to a topic in the user Kafka cluster, and the message delivery module is further configured to: determining a target theme according to the target theme identification carried in the log; determining a corresponding thread pool and a buffer queue according to the target subject; Storing the log into a corresponding buffer queue; And the producer thread responds to the existence of the log in the corresponding buffer queue, and delivers the log positioned at the head of the buffer queue to the corresponding user Kafka cluster. Optionally, the producer thread bulk delivers the log in the buffer queue based on a maximum delay time, a batch capacity upper limit, and a compression algorithm. Optionally, the delivery service further comprises: the monitoring module is used for sending index information and abnormal information in the log delivery process to the user side. Optionally, the monitori