CN-120956797-B - Internet of things data processing method and system based on large model

CN120956797BCN 120956797 BCN120956797 BCN 120956797BCN-120956797-B

Abstract

The application relates to the technical field of computers, in particular to a large-model-based data processing method and system of the Internet of things, wherein the method comprises the steps of acquiring multi-source data of the Internet of things through an edge node, processing the data of the Internet of things through a multi-source heterogeneous data processing method of Box-Cox conversion Z-score, and integrating the processed data of the Internet of things; the method comprises the steps of pruning and quantitatively compressing a preset large model to remove redundant or unimportant parameters in the large model and reduce the size and the calculated amount of the large model, obtaining problem description provided by a user, rapidly generating an ILP model frame through pruning and quantitatively compressing the large model according to the problem description, selecting a relay node for data transmission according to constraint conditions and objective functions, generating an optimal transmission path, and transmitting integrated Internet of things data to a cloud according to the optimal transmission path. The method and the device are beneficial to improving the data processing and transmission efficiency of the Internet of things.

Inventors

Hua Qinglin
Gong Yinlin
LI JUNLEI
LIU XIAOYAN

Assignees

长沙众微物联科技有限公司

Dates

Publication Date: 20260512
Application Date: 20251009

Claims (10)

1. The Internet of things data processing method based on the large model is characterized by comprising the following steps of: acquiring multi-source internet of things data through an edge node, processing the internet of things data through a multi-source heterogeneous data processing method of Box-Cox conversion Z-score, and integrating the processed internet of things data; pruning and quantitatively compressing a preset large model to remove redundant or unimportant parameters in the large model, so as to reduce the size and the calculated amount of the large model; Acquiring a problem description provided by a user, and rapidly generating an ILP model frame according to the problem description through pruning and quantizing the compressed large model, wherein the ILP model frame consists of constraint conditions and objective functions; Selecting a relay node for data transmission according to the constraint condition and the objective function, and generating an optimal transmission path; and transmitting the integrated data of the Internet of things to a cloud end according to the optimal transmission path.
2. The method for processing internet of things data based on a large model according to claim 1, wherein the pruning and quantization compressing the preset large model comprises: Acquiring an initial large model, training the initial large model, introducing a soft mask strategy and sparse factor cosine attenuation in the training process, and gradually pruning the large model; after training is completed, a preset large model is obtained; And converting the floating point number weight of the preset large model into a low-precision integer or semi-precision floating point number through quantization compression.
3. The method for processing data of internet of things based on big model according to claim 1, wherein the steps of obtaining the problem description provided by the user, and according to the problem description, rapidly generating the ILP model frame by pruning and quantizing the compressed big model comprise: Acquiring a problem description provided by a user, and converting the problem description into a structured logic expression through pruning and quantifying a compressed large model according to the problem description; Generating a logic rule according to the logic expression, wherein the logic rule is used for constructing constraint conditions and objective functions; and rapidly generating the ILP model framework according to the logic rules.
4. The method of claim 3, wherein the obtaining the problem description provided by the user, and converting the problem description into the structured logic expression by pruning and quantifying the compressed large model according to the problem description comprises: Acquiring a problem description provided by a user, and automatically defining a structured symbol vocabulary and a relation template through pruning and quantizing the compressed large model according to the problem description; And obtaining the structured logic expression according to the structured symbol vocabulary and the relation template.
5. The method for processing internet of things data based on the large model according to claim 1, wherein the acquiring the multi-source internet of things data through the edge node, processing the internet of things data through the multi-source heterogeneous data processing method of Box-Cox conversion Z-score, and integrating the processed internet of things data comprises: Acquiring multi-source internet of things data through an edge node, and converting the internet of things data into a form which is more similar to normal distribution through Box-Cox conversion to obtain first conversion data so as to reduce the problem of data migration of the internet of things; The first conversion data are converted into normal distribution with the mean value of 0 and the standard deviation of 1 through Z-score standardization, so that second conversion data are obtained, and the consistency of the multi-source internet of things data in dimension and magnitude order is ensured; And integrating the second conversion data.
6. The method for processing internet of things data based on a large model according to claim 1, wherein the method for processing the internet of things data by acquiring multi-source internet of things data through an edge node and processing the internet of things data through a multi-source heterogeneous data processing method of Box-Cox conversion Z-score, and after integrating the processed internet of things data, further comprises: And encrypting the integrated data of the Internet of things through an SM4 algorithm so as to improve the safety of data transmission.
7. The method for processing data of internet of things based on big model according to claim 1, wherein selecting a relay node for data transmission according to the constraint condition and the objective function, and generating an optimal transmission path comprises: Acquiring computing resources of an edge node, and judging whether integrated data of the Internet of things are transmitted through a relay node according to the computing resources; If yes, selecting a relay node for data transmission according to the constraint condition and the objective function, and generating an optimal transmission path; if not, the integrated data of the Internet of things are processed through the edge node.
8. The utility model provides a thing networking data processing device based on big model which characterized in that includes: The data integration module is used for acquiring multi-source internet of things data through the edge node, processing the internet of things data through a Box-Cox conversion Z-score multi-source heterogeneous data processing method, and integrating the processed internet of things data; the model compression module is used for pruning and quantitatively compressing a preset large model to remove redundant or unimportant parameters in the large model so as to reduce the size and the calculated amount of the large model; The model frame generation module is used for acquiring the problem description provided by the user, and rapidly generating an ILP model frame through pruning and quantizing the compressed large model according to the problem description, wherein the ILP model frame consists of constraint conditions and objective functions; The optimal path generation module is used for selecting a relay node for data transmission according to the constraint condition and the objective function and generating an optimal transmission path; and the data transmission module is used for transmitting the integrated data of the Internet of things to the cloud according to the optimal transmission path.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.

Description

Internet of things data processing method and system based on large model Technical Field The application relates to the technical field of computers, in particular to an Internet of things data processing method and system based on a large model. Background With the widespread use of internet of things (IoT) devices, the amount of data that they produce grows exponentially, which places higher demands on data processing and transmission. The traditional cloud computing mode is faced with the problems of high delay, insufficient bandwidth and the like when processing mass data, and particularly in scenes with high real-time requirements, such as intelligent transportation, industrial automation, telemedicine and the like. This mode relies on uploading data to a remote data center for processing, but as the amount of data increases, this centralized approach gradually exposes its limitations, particularly in terms of data transfer speed and response time. Therefore, how to efficiently process and transmit the data of the internet of things becomes one of the key points of the current research. Disclosure of Invention Based on the foregoing, it is necessary to provide a method and a system for processing data of the internet of things based on a large model, which can improve the data processing and transmission efficiency of the internet of things. In a first aspect, the present application provides a method for processing data of internet of things based on a large model, the method comprising: acquiring multi-source internet of things data through an edge node, processing the internet of things data through a multi-source heterogeneous data processing method of Box-Cox conversion Z-score, and integrating the processed internet of things data; pruning and quantitatively compressing a preset large model to remove redundant or unimportant parameters in the large model, so as to reduce the size and the calculated amount of the large model; Acquiring a problem description provided by a user, and rapidly generating an ILP model frame according to the problem description through pruning and quantizing the compressed large model, wherein the ILP model frame consists of constraint conditions and objective functions; Selecting a relay node for data transmission according to the constraint condition and the objective function, and generating an optimal transmission path; and transmitting the integrated data of the Internet of things to a cloud end according to the optimal transmission path. In one embodiment, pruning and quantitatively compressing the preset large model includes: Acquiring an initial large model, training the initial large model, introducing a soft mask strategy and sparse factor cosine attenuation in the training process, and gradually pruning the large model; after training is completed, a preset large model is obtained; And converting the floating point number weight of the preset large model into a low-precision integer or semi-precision floating point number through quantization compression. In one embodiment, the obtaining the problem description provided by the user, and according to the problem description, quickly generating the ILP model framework by pruning and quantifying the compressed large model includes: Acquiring a problem description provided by a user, and converting the problem description into a structured logic expression through pruning and quantifying a compressed large model according to the problem description; Generating a logic rule according to the logic expression, wherein the logic rule is used for constructing constraint conditions and objective functions; and rapidly generating the ILP model framework according to the logic rules. In one embodiment, the obtaining a problem description provided by a user, and converting the problem description into a structured logical expression by pruning and quantifying the compressed large model according to the problem description includes: Acquiring a problem description provided by a user, and automatically defining a structured symbol vocabulary and a relation template through pruning and quantizing the compressed large model according to the problem description; And obtaining the structured logic expression according to the structured symbol vocabulary and the relation template. In one embodiment, the acquiring the multi-source internet of things data through the edge node, and processing the internet of things data through a multi-source heterogeneous data processing method of Box-Cox conversion Z-score, and integrating the processed internet of things data includes: Acquiring multi-source internet of things data through an edge node, and converting the internet of things data into a form which is more similar to normal distribution through Box-Cox conversion to obtain first conversion data so as to reduce the problem of data migration of the internet of things; The first conversion data are converted into normal distribution