CN-121979738-A - Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method, a device, electronic equipment and a storage medium, wherein the method comprises the steps that a data processing unit obtains first log data, the data processing unit decomposes the first log data into first sub-data based on a minimum unit for decomposing the first log data, the first sub-data at least comprises partial data in the first log data, the minimum unit represents a minimum component in the first log data, the data processing unit performs data processing on the first sub-data based on a protocol included in the first sub-data to obtain second log data corresponding to the protocol, the protocol is prestored in the data processing unit, and the data processing unit controls a forwarding process of the second log data based on a protocol configuration item included in the data processing unit, wherein the protocol configuration item is used for determining a forwarding rule of the second log data.

Inventors

• Yang Daipi
• GE ZHEN

Assignees

• 联想(北京)有限公司

Dates

Publication Date

20260505

Application Date

20251217

Claims (10)

1. 1. A data processing method, comprising: the data processing unit obtains first log data; The data processing unit is used for decomposing the first log data into first sub-data based on a minimum unit for decomposing the first log data, wherein the first sub-data at least comprises partial data in the first log data; the data processing unit performs data processing on the first sub data based on the included protocol to obtain second log data corresponding to the protocol, wherein the protocol is stored in the data processing unit in advance; the data processing unit controls the forwarding process of the second log data based on the included protocol configuration item, wherein the protocol configuration item is used for determining the forwarding rule of the second log data.
2. 2. The method of claim 1, the method further comprising: determining a minimum unit for decomposing the first log data; the determining the minimum unit of the first log data for decomposition includes: determining a log type of the first log data; Determining a log splitting rule corresponding to the log type based on the log type, wherein the log splitting rule is preset based on a log format and a content attribute; And determining a minimum unit for decomposing the first log data based on the log splitting rule corresponding to the log type.
3. 3. The method of claim 1, wherein the performing data processing on the first sub-data based on the included protocol to obtain second log data corresponding to the protocol includes: The method comprises the steps of obtaining at least one protocol included in a data processing unit, wherein the protocol at least comprises a field data rule, a log format and a data reorganization rule, and the protocol is used for converting first sub-data into second log data conforming to the field data rule, the log format and the data reorganization rule; determining data to be processed from the first sub-data based on the field data rule of each protocol; and carrying out data recombination on the data to be processed based on the log format and the data recombination rule of each protocol to obtain second log data corresponding to each protocol.
4. 4. The method of claim 1, the data processing unit controlling a forwarding process of the second log data based on the included protocol configuration item, comprising: analyzing the protocol configuration item to obtain a forwarding rule of the second log data, wherein the forwarding rule at least comprises a forwarding mode, a target object and forwarding conditions; based on the forwarding rule, determining to-be-forwarded log data meeting the forwarding condition from the second log data; Transmitting the log data to be forwarded to a forwarding unit corresponding to the protocol, so that the forwarding unit forwards the log data to be forwarded to the target object in the forwarding mode.
5. 5. The method of claim 4, wherein the determining, based on the forwarding rule, log data to be forwarded that satisfies the forwarding condition from the second log data includes: Determining a first log type set in the forwarding condition, wherein the first log type is determined according to a forwarding rule pre-configured in a protocol configuration file; and determining the corresponding second log data as the log data to be forwarded in response to the log type of the second log data being matched with the first log type.
6. 6. The method of claim 1, the method further comprising: The data processing unit is used for receiving a configuration instruction of the protocol configuration items to obtain a first configuration file, wherein the first configuration file at least comprises protocol configuration items corresponding to all protocols; updating the protocol configuration items in the first configuration file based on the configuration instruction to obtain an updated first configuration file; the data processing unit loads the updated first configuration file to control the forwarding process of the second log data based on protocol configuration items included in the updated first configuration file.
7. 7. The method of claim 1, wherein the monitoring module of the data processing unit is instantiated in a single instance mode, and comprises a first monitoring module and a second monitoring module, wherein the first monitoring module is used for generating the first log data, and the second monitoring module is used for monitoring the change of the protocol configuration item.
8. 8. A data processing apparatus comprising: The acquisition module is used for acquiring the first log data by the data processing unit; the system comprises a data processing unit, a decomposition module and a data processing module, wherein the data processing unit is used for decomposing the first log data into first sub-data based on a minimum unit of the decomposition of the first log data; the processing module is used for carrying out data processing on the first sub data based on the included protocol by the data processing unit to obtain second log data corresponding to the protocol, wherein the protocol is stored in the data processing unit in advance; And the forwarding module is used for controlling the forwarding process of the second log data based on the included protocol configuration item, and the protocol configuration item is used for determining the forwarding rule of the second log data.
9. 9. An electronic device, comprising: And a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. 10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-7.

Description

Data processing method and device, electronic equipment and storage medium Technical Field The present application relates to computer technology, and in particular, to a data processing method, apparatus, electronic device, and storage medium. Background In the current data processing method, each protocol module independently processes log data, and log repetition processing exists. The data processing methods corresponding to the protocols are different, and have no unified management mode aiming at the protocols, when the format of log data is adjusted, the data processing method corresponding to the protocols needs to be modified, the code modification amount is large, and when a new protocol is imported, the data processing method corresponding to the protocol needs to be reconfigured. The existing data processing method has the problems of resource waste, complex protocol configuration, difficult maintenance and poor expansibility. Disclosure of Invention The embodiment of the application provides a data processing method, a data processing device, electronic equipment and a storage medium. The application provides a data processing method, which comprises the steps that a data processing unit obtains first log data, the data processing unit decomposes the first log data into first sub-data based on a minimum unit for decomposing the first log data, the first sub-data at least comprises partial data in the first log data, the minimum unit characterizes the minimum component in the first log data, the data processing unit processes the first sub-data based on an included protocol to obtain second log data corresponding to the protocol, the protocol is prestored in the data processing unit, the data processing unit controls the forwarding process of the second log data based on an included protocol configuration item, and the protocol configuration item is used for determining the forwarding rule of the second log data. According to one embodiment of the application, the method further comprises the steps of determining the minimum unit for decomposing the first log data, wherein the step of determining the minimum unit for decomposing the first log data comprises the steps of determining the log type of the first log data, determining a log splitting rule corresponding to the log type based on the log type, wherein the log splitting rule is preset based on a log format and a content attribute, the minimum units corresponding to different log splitting rules are different, and the minimum unit for decomposing the first log data is determined based on the log splitting rule corresponding to the log type. According to one embodiment of the application, the data processing is performed on the first sub-data based on the included protocol to obtain second log data corresponding to the protocol, wherein the data processing unit comprises at least one protocol which is included in the data processing unit and at least comprises a field data rule, a log format and a data reorganization rule, the protocol is used for converting the first sub-data into the second log data which accords with the field data rule, the log format and the data reorganization rule, the data to be processed is determined from the first sub-data based on the field data rule of each protocol, and the data to be processed is reorganized based on the log format and the data reorganization rule of each protocol to obtain the second log data corresponding to each protocol. According to one embodiment of the application, the data processing unit controls the forwarding process of the second log data based on the included protocol configuration item, and comprises a forwarding rule for analyzing the protocol configuration item to obtain the second log data, wherein the forwarding rule at least comprises a forwarding mode, a target object and a forwarding condition, determines log data to be forwarded meeting the forwarding condition from the second log data based on the forwarding rule, and transmits the log data to be forwarded to a forwarding unit corresponding to the protocol, so that the forwarding unit forwards the log data to be forwarded to the target object in the forwarding mode. According to one embodiment of the application, the method for determining the log data to be forwarded, which meets the forwarding condition, from the second log data based on the forwarding rule comprises the steps of determining a first log type set in the forwarding condition, wherein the first log type is determined according to a forwarding rule preset in a protocol configuration file, forwarding modes and target objects corresponding to different log types are different, and determining the corresponding second log data as the log data to be forwarded in response to the fact that the log type of the second log data is matched with the first log type. According to one embodiment of the application, the method further comprises the steps of responding t