Search

CN-122020708-A - Log desensitization method, device, equipment and medium

CN122020708ACN 122020708 ACN122020708 ACN 122020708ACN-122020708-A

Abstract

The application belongs to the field of big data, and relates to a log desensitizing method, device, equipment and medium. And acquiring a plurality of log lines to be desensitized based on the output directory path, generating a line sequence number of each log line to be desensitized through an atomic counter, assembling a plurality of strip sequence number messages with each log line to be desensitized, and putting the strip sequence number messages into a blocking queue determined by a queue size parameter. And creating a thread pool based on the number of concurrent threads, reading the message from the blocking queue by the threads, replacing sensitive data according to a desensitization rule to obtain the message containing the line sequence number and the desensitization log line, and finally orderly writing the desensitization log into a desensitization log file according to the line sequence number. The application can be applied to the business fields of finance, science, insurance, medical treatment and the like, and can solve the problem that the data is difficult to locate and trace after desensitization.

Inventors

  • ZHAO CHUXUAN
  • ZHANG XIAOLONG

Assignees

  • 平安科技（深圳）有限公司

Dates

Publication Date
20260512
Application Date
20260109

Claims (10)

  1. 1. A method of desensitizing a log comprising the steps of: acquiring concurrent thread numbers, desensitization rules, queue size parameters and output directory paths from configuration files of log desensitization programs; Based on the output directory path, acquiring a plurality of log lines to be desensitized, and generating a line sequence number of each log line to be desensitized through an atomic counter; Assembling each log line to be desensitized and the line sequence numbers to generate a plurality of strip sequence number messages, and placing the strip sequence number messages into a blocking queue, wherein the accommodation quantity of the blocking queue is determined by the queue size parameter; creating a thread pool based on the number of concurrent threads, and reading the plurality of strip sequence number messages from the blocking queue through each thread in the thread pool; based on the desensitization rule, carrying out sensitive data replacement on the read multi-strip serial number message to obtain a plurality of desensitized messages, wherein each desensitized message comprises the line serial number and a desensitized log line; and writing the desensitization log lines into a desensitization log file orderly based on the line serial numbers.
  2. 2. The method according to claim 1, wherein the step of obtaining a plurality of log lines to be desensitized based on the output directory path comprises: when the log desensitization program is started, detecting whether a desensitization log file exists in a target log directory or not based on the output directory path; If the desensitization log file exists in the target log catalog, acquiring the latest timestamp from the desensitization log file; obtaining a log file from a source log catalog, wherein the log file comprises a plurality of log lines; extracting modification time corresponding to each log line in the log file; and determining the log line with the modification time longer than the latest timestamp as the log line to be desensitized.
  3. 3. The method according to claim 2, wherein the step of obtaining the latest timestamp from the desensitized log file if the desensitized log file exists in the target log directory, specifically comprises: If the desensitization log file exists in the target log catalog, reading metadata information of the desensitization log file; extracting time stamp data from the metadata information, the time stamp data comprising a plurality of time records; And sequencing the plurality of time records based on the time sequence to obtain a time stamp sequence, and determining the latest time stamp of the desensitization log file according to the time stamp sequence.
  4. 4. The method according to claim 1, wherein the step of generating a line number for each log line to be desensitized by an atomic counter comprises: initializing an atomic counter, wherein the initial value of the atomic counter is set to be a preset value; and triggering the self-increment operation of the atomic counter based on a preset value for each log line to be desensitized to obtain a corresponding line sequence number.
  5. 5. The method according to claim 1, wherein the step of replacing sensitive data of the read message with sequence number based on the desensitization rule to obtain a desensitized message specifically comprises: analyzing the read message with the sequence number according to a preset identification rule, and determining sensitive data; classifying the sensitive data according to the desensitization rule, and determining sensitive data of multiple sensitive categories and replacement modes of each sensitive category; And adopting the replacement mode to perform replacement operation on the sensitive data to obtain a desensitized message.
  6. 6. The method according to claim 1, wherein the step of orderly writing the desensitized log lines into a desensitized log file based on the line sequence number, comprises: based on the line sequence number, placing the desensitized log line into a priority queue, and sequencing the desensitized log line by the priority queue according to the sequence number of the line sequence number; And reading the desensitization log lines in the priority queue according to the sequence of the line serial numbers by a writing thread to obtain orderly desensitization log line data, and writing the orderly desensitization log line data into a desensitization log file.
  7. 7. The method of claim 1, further comprising, after the step of writing the desensitized log lines in order into a desensitized log file based on the line sequence number: monitoring the log size and the writing interval of the desensitized log file in real time; Stopping writing data into the desensitized log file when the log size of the desensitized log file is greater than or equal to a preset size threshold or the writing interval is greater than or equal to a preset time interval; When a new desensitization log line is detected, generating a new desensitization log file; And writing the new desensitization log lines into the new desensitization log file in sequence through a writing thread based on the line serial numbers corresponding to the new desensitization log lines.
  8. 8. A log desensitizing apparatus, comprising: the first acquisition module is used for acquiring the concurrent thread number, the desensitization rule, the queue size parameter and the output directory path from the configuration file of the log desensitization program; the second acquisition module is used for acquiring a plurality of log lines to be desensitized based on the output directory path, and generating a line sequence number of each log line to be desensitized through an atomic counter; The assembling module is used for assembling each log line to be desensitized and the line sequence numbers, generating a plurality of band sequence number messages, and placing the band sequence number messages into a blocking queue, wherein the accommodation quantity of the blocking queue is determined by the queue size parameter; The reading module is used for creating a thread pool based on the concurrent thread number, and reading the plurality of band sequence number messages from the blocking queue through each thread in the thread pool; The replacing module is used for replacing the read multi-strip serial number message with sensitive data based on the desensitization rule to obtain a plurality of desensitized messages, wherein each desensitized message comprises the line serial number and a desensitization log line; And the first writing module is used for orderly writing the desensitization log lines into a desensitization log file based on the line serial numbers.
  9. 9. A computer device comprising a memory having stored therein computer readable instructions which when executed by a processor implement the steps of the log desensitization method according to any of claims 1-7.
  10. 10. A computer readable storage medium having stored thereon computer readable instructions which when executed by a processor implement the steps of the log desensitization method according to any of claims 1-7.

Description

Log desensitization method, device, equipment and medium Technical Field The application relates to the field of big data, and is applied to online processing business scenes of financial science and technology, insurance, medical treatment and the like, in particular to a log desensitizing method, device, equipment and medium. Background With the perfection of data security regulations, enterprise data processing compliance requirements are becoming increasingly stringent. The log is used as a core carrier for recording the operation and user operation of the system, contains sensitive data such as an identity card number, a bank card number, a mobile phone number and the like, and needs to realize safety protection through desensitization treatment. In practical application, the audit log has particularly outstanding desensitization requirements, such as account operation log of a financial transaction system, medical record access log of a medical system and insurance claim settlement process log, and the logs not only contain sensitive information, but also need to meet compliance traceability requirements. However, the existing desensitization technology has obvious defects that firstly, the association mark of a desensitization log and an original log is not available, the desensitization log cannot be positioned through the original information, the tracing is affected, secondly, full processing is adopted, resources are wasted by repeated calculation, the log sequence is disordered easily caused by multithreading, the audit log timeliness is destroyed, thirdly, no configured resource management and control is available, a data buffering mechanism is lacked, and the processing stability is poor. In summary, the prior art has the problems of difficult data tracing after desensitization, low incremental processing efficiency, disordered log sequence, insufficient stability and the like. Disclosure of Invention The embodiment of the application aims to provide a log desensitizing method, a device, computer equipment and a storage medium, which are used for solving the problems of difficult data tracing, low incremental processing efficiency, disordered log sequence and insufficient stability of the existing log desensitizing technology. In a first aspect, a log desensitizing method is provided, and the following technical scheme is adopted: The method comprises the steps of obtaining the concurrent thread number, a desensitization rule, queue size parameters and an output directory path from a configuration file of a log desensitization program, obtaining a plurality of log lines to be desensitized based on the output directory path, generating a line sequence number of each log line to be desensitized through an atomic counter, assembling each log line to be desensitized and each line sequence number to generate a plurality of strip sequence number messages, placing the plurality of strip sequence number messages into a blocking queue, wherein the accommodation number of the blocking queue is determined by the queue size parameters, creating a thread pool based on the concurrent thread number, reading the plurality of strip sequence number messages from the blocking queue through each thread in the thread pool, replacing sensitive data of the read plurality of strip sequence number messages based on the desensitization rule to obtain a plurality of desensitized messages, wherein each desensitized message comprises a line sequence number and a desensitized log line, and orderly writing the desensitized log lines into the desensitization log file based on the line sequence numbers. In a second aspect, a log desensitizing device is provided, which adopts the following technical scheme: the first acquisition module is used for acquiring the concurrent thread number, the desensitization rule, the queue size parameter and the output directory path from the configuration file of the log desensitization program; The second acquisition module is used for acquiring a plurality of log lines to be desensitized based on the output directory path, and generating a line sequence number of each log line to be desensitized through an atomic counter; The assembly module is used for assembling each log line and each line sequence number to be desensitized, generating a plurality of strip sequence number messages, and placing the plurality of strip sequence number messages into a blocking queue, wherein the accommodation quantity of the blocking queue is determined by the queue size parameter; The reading module is used for creating a thread pool based on the number of concurrent threads, and reading the multi-band serial number message from the blocking queue through each thread in the thread pool; The replacing module is used for replacing the sensitive data of the read multi-strip serial number message based on the desensitization rule to obtain a plurality of desensitized messages, wherein each desensitized message co