Search

CN-121981277-A - Sensor data enhanced multi-mode Internet of things large model reasoning method

CN121981277ACN 121981277 ACN121981277 ACN 121981277ACN-121981277-A

Abstract

The invention provides a multi-mode Internet of things large model reasoning method for enhancing sensor data, which relates to the technical field of computers, and comprises the following steps of 1, acquiring multi-mode sensor original data acquired by Internet of things terminal equipment, preprocessing the multi-mode sensor original data, and obtaining preprocessed sensor data; the method comprises the steps of (1) selecting physical deployment positions of gateway equipment of the Internet of things in an actual deployment environment corresponding to the preprocessed sensor data to construct a space reference configuration, resolving the space reference configuration to obtain two-dimensional pose features and relative motion features of the gateway equipment of the Internet of things in a plane, integrating the two-dimensional pose features and the relative motion features to form two-dimensional topological data, performing granularity refinement operation on the space reference configuration based on the two-dimensional topological data to form a plurality of discrete areas, and obtaining space correction parameters according to topological distribution features and structural gradient changes of the discrete areas. The invention realizes the self-adaptive enhancement of the sensor data.

Inventors

  • YE WEIKANG
  • ZHANG YINFENG
  • SUN KE
  • WU ZHIHUI

Assignees

  • 厦门天堉物联网科技有限公司

Dates

Publication Date
20260505
Application Date
20260407

Claims (10)

  1. 1. The multi-mode Internet of things large model reasoning method with enhanced sensor data is characterized by comprising the following steps: Step 1, acquiring original data of a multi-mode sensor acquired by terminal equipment of the Internet of things, and preprocessing the original data of the multi-mode sensor to obtain preprocessed sensor data; Step 2, selecting a physical deployment position of the gateway equipment of the Internet of things in an actual deployment environment corresponding to the preprocessed sensor data to construct a space reference configuration; step 3, performing granularity refinement operation on the space reference configuration based on the two-dimensional topological data to form a plurality of discrete areas; Step 4, performing space self-adaptive enhancement processing on the original data of the multi-modal sensor through the space correction parameters to obtain enhanced multi-modal sensor data; step 5, inputting the enhanced multi-mode sensor data into a preset encoder for feature extraction and encoding to obtain sensor feature representation; And 6, carrying out feature fusion on the sensor feature representation and a main network of the multi-mode large model, and driving the multi-mode large model to execute reasoning calculation based on the fused features to obtain a reasoning result.
  2. 2. The sensor data enhanced multi-mode internet of things large model reasoning method of claim 1, wherein the method is characterized by obtaining multi-mode sensor raw data collected by internet of things terminal equipment, preprocessing the multi-mode sensor raw data to obtain preprocessed sensor data, and comprises the following steps: and receiving the original data of the multi-mode sensor uploaded by the distributed Internet of things terminal equipment through the Internet of things communication protocol, and executing time stamp alignment and data integrity check on the original data of the multi-mode sensor to obtain the preprocessed sensor data.
  3. 3. The sensor data enhanced multi-mode Internet of things large model reasoning method of claim 2 is characterized in that a spatial reference configuration is constructed by selecting physical deployment positions of Internet of things gateway equipment in an actual deployment environment corresponding to the preprocessed sensor data, the spatial reference configuration is solved to obtain two-dimensional pose features and relative motion features of the in-plane Internet of things gateway equipment, and the two-dimensional topological data are formed by integration, and the method comprises the following steps: Based on the actual deployment environment area identified by the preprocessed sensor data, the physical deployment position coordinates and the real-time state information of all the gateway devices of the Internet of things in the area are called from an environment topology database to form a gateway device position set; screening out expansion gateway equipment with signal interaction relation with three reference gateways in an initial space reference configuration through a gateway equipment position set, and integrating physical deployment position coordinates of the reference gateways and the expansion gateways together to construct a complete space reference configuration; Mapping the three-dimensional space coordinates to a two-dimensional plane according to the coordinate distribution of each gateway device in the complete space reference configuration to obtain the position coordinates, the orientation angles and the coverage outlines of each gateway device in the plane, so as to form two-dimensional pose features; according to the two-dimensional pose characteristics, analyzing the relative distance change trend and the overlapping dynamic of the coverage areas between adjacent gateway devices, extracting the relative displacement vector and the overlapping rate time sequence change characteristics between the gateway devices, and forming relative motion characteristics; based on the two-dimensional pose features and the relative motion features, an adjacency matrix and weight data representing the spatial relationship of gateway equipment are constructed, so that two-dimensional topological data are obtained.
  4. 4. The method for reasoning the large model of the multi-mode Internet of things with enhanced sensor data according to claim 3, wherein the method for reasoning the large model of the multi-mode Internet of things with enhanced sensor data is characterized in that granularity refinement operation is carried out on a spatial reference configuration based on two-dimensional topological data to form a plurality of discrete areas, and the method for deriving the spatial correction parameters according to topological distribution characteristics and structural gradient changes of the discrete areas comprises the following steps: Analyzing an adjacency matrix and weight data representing the spatial relationship of gateway equipment in the two-dimensional topological data according to the two-dimensional topological data, and extracting the connection strength and coverage density distribution characteristics among gateway nodes; Based on the distribution characteristics of the connection strength and the coverage density, performing self-adaptive granularity refinement operation on the complete space reference configuration, adopting fine granularity subdivision in the area with high connection strength and large coverage density, and adopting coarse granularity subdivision in the connection sparse area to obtain a plurality of discrete areas matched with the topological structure; Based on the spatial distribution of a plurality of discrete areas, calculating the topological connectivity and coverage overlap ratio between adjacent areas to obtain topological distribution characteristics representing the spatial correlation between the areas; Calculating a numerical gradient vector along the boundary direction of the region according to the numerical variation trend of the sensor data in the discrete regions, and analyzing the continuity variation of the gradient amplitude and the direction to obtain a structural gradient variation characteristic representing the distortion degree of the spatial structure; Based on the topological distribution characteristic and the structural gradient change characteristic, the spatial position offset, the signal attenuation compensation coefficient and the time synchronization correction value corresponding to each discrete region are comprehensively calculated to obtain the spatial correction parameters.
  5. 5. The method for reasoning the large model of the multi-modal internet of things with enhanced sensor data according to claim 4, wherein the method for performing spatial adaptive enhancement processing on the original data of the multi-modal sensor through the spatial correction parameters to obtain the enhanced multi-modal sensor data comprises the following steps: according to the space correction parameters and the sensor data after verification, carrying out offset compensation on original space coordinates of each data point in the sensor data after verification according to space position offset contained in the space correction parameters to obtain sensor data after space position correction; based on the sensor data after the spatial position correction, carrying out self-adaptive gain adjustment on the signal intensity value of each data point according to the signal attenuation compensation coefficient in the spatial correction parameter, and compensating the signal attenuation caused by the propagation distance and the environmental interference to obtain the sensor data after the signal intensity is enhanced; Based on the sensor data with enhanced signal strength, microsecond level alignment correction is carried out on time stamps of sensor data of different modes according to time synchronization correction values in the space correction parameters, so that time sequence drift among multi-source heterogeneous sensors is eliminated, and multi-mode sensor data after time sequence synchronization is obtained; Based on the multi-mode sensor data after time sequence synchronization, according to the spatial distribution characteristics of a plurality of discrete areas, data complement is carried out in a data sparse area, local noise is restrained in a data dense area, and the enhanced multi-mode sensor data with uniform spatial distribution and noise restraint is obtained.
  6. 6. The sensor data enhanced multi-modal internet of things large model reasoning method of claim 5, wherein step 5 comprises: Receiving the enhanced multi-mode sensor data, and respectively performing normalization and standardization processing on the temperature, humidity, vibration, image and audio mode data according to the data type and dimension characteristics of each mode sensor to obtain preprocessed multi-mode data; based on the preprocessed multi-mode data, a mode special feature extraction module in a preset encoder is called, convolution filtering is respectively carried out on each mode data, and local feature vectors and global statistical features of each mode are extracted to form a multi-mode original feature set; According to the multi-modal original feature set, spatial alignment is carried out on the same-modal features acquired at different spatial positions through the spatial relationship of the gateway equipment represented in the two-dimensional topological data, and time sequence synchronous alignment is carried out on different-modal features acquired at the same position, so that a multi-modal feature sequence with spatial and time sequence alignment is obtained; Calculating the correlation weight among the modal features through a spatial and time sequence aligned multi-modal feature sequence and a cross-modal attention mechanism in a preset encoder, carrying out weighted fusion on the high-correlation modal features, and carrying out complementary enhancement on the low-correlation modal features to obtain a fused cross-modal joint feature; according to the fused cross-modal joint characteristics, dimension reduction and semantic abstraction are carried out through a characteristic compression module in a preset encoder, key characteristic components related to reasoning tasks are reserved, redundant noise components are restrained, and sensor characteristic representation adapting to the multi-modal large-model input interface is obtained.
  7. 7. The sensor data enhanced multi-mode internet of things large model reasoning method of claim 6, wherein feature fusion is performed on the sensor feature representation and a main network of the multi-mode large model, reasoning calculation is performed on the basis of the fused feature-driven multi-mode large model, and a reasoning result is obtained, and the method comprises the following steps: Receiving sensor characteristic representation, extracting visual and language joint characteristics output by a backbone network in an intermediate layer from a preloaded multi-mode large model, and forming backbone characteristics to be fused; Based on the dimension difference between the sensor feature representation and the trunk feature, performing linear projection transformation on the sensor feature representation to enable the dimension difference of the feature to be matched with the trunk feature, and performing space alignment on the two types of features according to a semantic similarity calculation result to obtain a feature pair to be fused, wherein the dimension of the feature pair is consistent and the semantic of the feature pair is aligned; Based on the feature pair to be fused, carrying out weighted fusion on the two types of features according to the contribution weight of the trunk feature, and generating a fused enhancement feature; Based on the fused enhancement features, the fused enhancement features are used as condition input to be injected into an inference layer of the multi-mode large model, the large model is driven to execute forward propagation calculation, and intermediate inference results comprising target identification, state prediction and anomaly diagnosis are generated through an attention interaction layer, a semantic inference layer and a decision output layer in sequence; Based on the intermediate reasoning result, the result verification and the confidence correction are carried out by combining the spatial context information in the two-dimensional topological data, the result propagation enhancement is carried out on the low-confidence region according to the spatial adjacency, and finally the reasoning result with spatial interpretability is output.
  8. 8. A sensor data enhanced multi-modal internet of things large model reasoning system implementing the method of any of claims 1 to 7, comprising: The acquisition module is used for acquiring the original data of the multi-mode sensor acquired by the terminal equipment of the Internet of things, preprocessing the original data of the multi-mode sensor and obtaining preprocessed sensor data; the integration module is used for selecting the physical deployment position of the gateway equipment of the Internet of things in the actual deployment environment corresponding to the preprocessed sensor data to construct a space reference configuration; The calculation module is used for performing granularity refinement operation on the space reference configuration based on the two-dimensional topological data to form a plurality of discrete areas; The enhancement module is used for carrying out space self-adaptive enhancement processing on the original data of the multi-mode sensor through the space correction parameters to obtain enhanced multi-mode sensor data; the extraction module is used for inputting the enhanced multi-mode sensor data into a preset encoder to perform feature extraction and encoding to obtain sensor feature representation; and the fusion module is used for carrying out feature fusion on the sensor feature representation and the main network of the multi-mode large model, and carrying out reasoning calculation on the basis of the fused feature-driven multi-mode large model to obtain a reasoning result.
  9. 9. A computing device, comprising: One or more processors; Storage means for storing one or more programs which when executed by the one or more processors cause the one or more processors to implement the method of any of claims 1to 7.
  10. 10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a program which, when executed by a processor, implements the method according to any of claims 1 to 7.

Description

Sensor data enhanced multi-mode Internet of things large model reasoning method Technical Field The invention relates to the technical field of computers, in particular to a multi-mode Internet of things large model reasoning method with enhanced sensor data. Background Under the development background of deep fusion of the Internet of things and the large model, the multi-mode Internet of things large model is widely applied to scenes such as industrial monitoring, intelligent parks and intelligent security, and becomes a core technology for realizing equipment state sensing, anomaly diagnosis and scene reasoning decision, but the reasoning process is highly dependent on the quality and feature fusion efficiency of front-end sensor data. Taking an intelligent garden internet of things monitoring system as an example, the system distributes multi-mode sensors such as temperature and humidity, vibration, vision, audio and the like in a garden, collects multi-source data such as operation, environmental change, personnel activities and the like of equipment in the garden, relies on the multi-mode large model to develop reasoning work of equipment fault early warning, environmental risk research and judgment and abnormal behavior recognition, the sensor raw data which is not processed in a targeted manner is input into the large model, spatial topological features of physical deployment of an internet of things gateway are not combined to correct and enhance the data, so that the sensor data has the problems of spatial position offset, signal attenuation, multi-source heterogeneous data time sequence drift and the like, the spatial distribution density of the data is uneven, local noise interference is obvious, meanwhile, the sensor features are only simply extracted and then are directly input into the large model, dimensional matching and semantic alignment depth fusion are not carried out on main network features of the large model, and the reasoning result is not verified and corrected by combining spatial context information, and finally the large model is caused to have insufficient utilization of the spatial features of the internet of things scene, and poor accuracy and interpretability of the reasoning result. Disclosure of Invention The invention aims to solve the technical problem of providing a multi-mode Internet of things large model reasoning method for enhancing sensor data, and realizing self-adaptive enhancement of the sensor data. In order to solve the technical problems, the technical scheme of the invention is as follows: in a first aspect, a method for reasoning a large model of a multi-mode internet of things with enhanced sensor data, the method comprising: Step 1, acquiring original data of a multi-mode sensor acquired by terminal equipment of the Internet of things, and preprocessing the original data of the multi-mode sensor to obtain preprocessed sensor data; Step 2, selecting a physical deployment position of the gateway equipment of the Internet of things in an actual deployment environment corresponding to the preprocessed sensor data to construct a space reference configuration; step 3, performing granularity refinement operation on the space reference configuration based on the two-dimensional topological data to form a plurality of discrete areas; Step 4, performing space self-adaptive enhancement processing on the original data of the multi-modal sensor through the space correction parameters to obtain enhanced multi-modal sensor data; step 5, inputting the enhanced multi-mode sensor data into a preset encoder for feature extraction and encoding to obtain sensor feature representation; And 6, carrying out feature fusion on the sensor feature representation and a main network of the multi-mode large model, and driving the multi-mode large model to execute reasoning calculation based on the fused features to obtain a reasoning result. Further, acquiring the original data of the multi-mode sensor acquired by the terminal equipment of the internet of things, preprocessing the original data of the multi-mode sensor to obtain preprocessed sensor data, wherein the method comprises the following steps: and receiving the original data of the multi-mode sensor uploaded by the distributed Internet of things terminal equipment through the Internet of things communication protocol, and executing time stamp alignment and data integrity check on the original data of the multi-mode sensor to obtain the preprocessed sensor data. Further, a spatial reference configuration is constructed by selecting the physical deployment position of the gateway equipment of the Internet of things in the actual deployment environment corresponding to the preprocessed sensor data, the spatial reference configuration is calculated to obtain the two-dimensional pose characteristics and the relative motion characteristics of the gateway equipment of the Internet of things in the plane, and the two-dimensional top