CN-122023985-A - Target monitoring method, device, equipment, storage medium and program product

CN122023985ACN 122023985 ACN122023985 ACN 122023985ACN-122023985-A

Abstract

The invention discloses a target monitoring method, a device, equipment, a storage medium and a program product, wherein multi-mode data are acquired in real time through multi-sensor equipment which is deployed in a target area in advance; the method comprises the steps of carrying out feature extraction and feature fusion processing on acquired multi-mode data to obtain data features, constructing a knowledge graph related to the field of target monitoring, extracting semantic features of the knowledge graph, fusing the data features and the semantic features to generate feature vectors combining quantized data and semantic knowledge, recording the feature vectors as target feature vectors, and carrying out target monitoring operation according to the target feature vectors and the knowledge graph. By adopting the method and the device, the data features acquired by the multiple sensors and the semantic features acquired based on the knowledge graph can be subjected to deep fusion, a complete target monitoring process is finished cooperatively, and the accuracy and the intellectualization of target monitoring are improved.

Inventors

WANG YAJUN
ZHU JIANHAO
DONG XING
LI YANYAN
LI ZHENGTAO

Assignees

中国移动通信集团河南有限公司
中国移动通信集团有限公司

Dates

Publication Date: 20260512
Application Date: 20260128

Claims (12)

1. A method of target monitoring, comprising: acquiring multi-mode data in real time through multi-sensor equipment deployed in a target area in advance; Performing feature extraction and feature fusion processing on the acquired multi-mode data to obtain data features; Constructing a knowledge graph related to the target monitoring field; extracting semantic features of the knowledge graph; Fusing the data features and the semantic features to generate feature vectors combining quantized data and semantic knowledge, and marking the feature vectors as target feature vectors; And performing target monitoring operation according to the target feature vector and the knowledge graph.
2. The method for monitoring a target according to claim 1, wherein the performing feature extraction and feature fusion processing on the collected multi-modal data to obtain data features includes: extracting the characteristics of each mode data in the multi-mode data; evaluating a confidence level of each modality data based on the modality quality factor; calculating a weight coefficient of each mode data according to the confidence coefficient; And carrying out weighted fusion on the characteristics of each mode of data according to the weight coefficient to obtain fused data characteristics.
3. The method of object monitoring as set forth in claim 2, wherein the multi-sensor device comprises at least one of an optical camera, a radar, an acoustic wave sensor, and an infrared sensor, and wherein the multi-modality data comprises at least one of image data, radar data, acoustic wave data, and infrared data.
4. The method for monitoring a target according to claim 1, wherein the constructing a knowledge graph related to the field of target monitoring comprises: Constructing a knowledge graph related to the target monitoring field according to a preset graph database, wherein the knowledge graph comprises a plurality of nodes related to the target monitoring field and edges for representing association relations among the nodes; And updating corresponding nodes and edges in the knowledge graph in real time according to the multi-mode data to obtain a dynamic knowledge graph.
5. The method for monitoring targets according to claim 4, wherein updating corresponding nodes and edges in the knowledge graph in real time according to the multi-modal data to obtain a dynamic knowledge graph comprises: When the multi-mode data is judged to meet the preset updating triggering condition, updating the corresponding nodes and edges in the knowledge graph according to a message transmission mechanism of the graph neural network; Generating natural language description according to a preset prompting word template and the multi-modal data; performing semantic speculation on the natural language description by adopting a large language model to generate a new triplet and a corresponding confidence coefficient, wherein the triplet is in an entity-relation-entity format; and adding a new triplet with the confidence coefficient larger than a preset confidence coefficient threshold value into the knowledge graph, and updating the knowledge graph.
6. The method of claim 1, wherein the fusing the data features and the semantic features to generate feature vectors combining quantized data and semantic knowledge, denoted as target feature vectors, comprises: And splicing the data features and the semantic features along the same dimension by adopting Concatenate algorithm to generate a high-latitude feature vector combining the quantized data and the semantic knowledge, and marking the high-latitude feature vector as a target feature vector.
7. The target monitoring method of claim 1, wherein the target monitoring operation comprises target identification, behavioral analysis, risk assessment, and policy generation; The target monitoring operation according to the target feature vector and the knowledge graph comprises the following steps: inputting the target feature vector and the knowledge graph to a preset target recognition model to obtain target class probability output by the target recognition model, and determining a recognized target according to the target class probability; Inputting the target feature vector and the knowledge graph to a preset behavior analysis model to obtain a behavior tag sequence output by the behavior analysis model; inputting the behavior tag sequence and the knowledge graph to a preset risk assessment model to obtain a risk value output by the risk assessment model; and inputting the risk value and the knowledge graph to a preset decision generation model to obtain an action vector output by the decision generation model as a treatment strategy.
8. The method of claim 7, wherein the target recognition model adopts a network structure of a full-connection layer FC and a Softmax classifier, the behavior analysis model adopts a network structure of a long-short-term memory neural network LSTM and an attention mechanism, the risk assessment model adopts a network structure of a full-connection layer FC and a Sigmoid function, and the decision generation model adopts a strategy network structure.
9. A target monitoring apparatus, comprising: the multi-mode data acquisition module is used for acquiring multi-mode data in real time through multi-sensor equipment deployed in a target area in advance; the data feature extraction module is used for carrying out feature extraction and feature fusion processing on the acquired multi-mode data to obtain data features; the knowledge graph construction module is used for constructing a dynamic knowledge graph related to the target monitoring field; the semantic feature extraction module is used for extracting semantic features of the dynamic knowledge graph; The feature vector fusion module is used for fusing the data features and the semantic features to generate feature vectors combining quantized data and semantic knowledge, and marking the feature vectors as target feature vectors; and the target monitoring module is used for carrying out target monitoring operation according to the target feature vector and the knowledge graph.
10. An object monitoring device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the object monitoring method according to any one of claims 1 to 8 when executing the computer program.
11. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored computer program, wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the object monitoring method according to any one of claims 1 to 8.
12. A computer program product, characterized in that the computer program product comprises a computer program or computer instructions which, when executed by a processor, implements the object monitoring method according to any of claims 1 to 8.

Description

Target monitoring method, device, equipment, storage medium and program product Technical Field The present invention relates to the field of artificial intelligence technology, and in particular, to a target monitoring method, apparatus, device, storage medium, and program product. Background The target monitoring technology has application requirements in various field scenes, such as low-altitude flight field, ground security field, industrial operation and maintenance field, urban management field and the like. Currently, a target monitoring system commonly adopts a multi-sensor fusion and static knowledge graph technology to realize target monitoring, acquires data through multiple devices such as a radar, an optical camera, an infrared sensor and the like, integrates multi-mode information through traditional weighted average or decision-level fusion (such as a voting method) to realize target identification, and performs behavior prediction and the like through pre-defined rules of a pre-constructed static knowledge graph. However, since the multi-sensor data fusion and the knowledge graph query process are independent of each other, behavior prediction deviation is easily caused, and accurate and effective processing decisions are not generated. Disclosure of Invention The embodiment of the invention aims to provide a target monitoring method, device, equipment, storage medium and program product, which can carry out deep fusion on data features acquired by multiple sensors and semantic features acquired based on a knowledge graph, complete a target monitoring flow cooperatively and improve the accuracy and the intellectualization of target monitoring. In order to achieve the above object, an embodiment of the present invention provides a target monitoring method, including: acquiring multi-mode data in real time through multi-sensor equipment deployed in a target area in advance; Performing feature extraction and feature fusion processing on the acquired multi-mode data to obtain data features; Constructing a knowledge graph related to the target monitoring field; extracting semantic features of the knowledge graph; Fusing the data features and the semantic features to generate feature vectors combining quantized data and semantic knowledge, and marking the feature vectors as target feature vectors; And performing target monitoring operation according to the target feature vector and the knowledge graph. As an improvement of the above scheme, the feature extraction and feature fusion processing are performed on the acquired multi-mode data to obtain data features, including: extracting the characteristics of each mode data in the multi-mode data; evaluating a confidence level of each modality data based on the modality quality factor; calculating a weight coefficient of each mode data according to the confidence coefficient; And carrying out weighted fusion on the characteristics of each mode of data according to the weight coefficient to obtain fused data characteristics. As an improvement of the above, the multi-sensor device includes at least one of an optical camera, a radar, an acoustic wave sensor, and an infrared sensor, and the multi-modal data includes at least one of image data, radar data, acoustic wave data, and infrared data. As an improvement of the above solution, the construction of a knowledge graph related to the target monitoring field includes: Constructing a knowledge graph related to the target monitoring field according to a preset graph database, wherein the knowledge graph comprises a plurality of nodes related to the target monitoring field and edges for representing association relations among the nodes; And updating corresponding nodes and edges in the knowledge graph in real time according to the multi-mode data to obtain a dynamic knowledge graph. As an improvement of the above solution, the updating corresponding nodes and edges in the knowledge graph in real time according to the multi-mode data to obtain a dynamic knowledge graph includes: When the multi-mode data is judged to meet the preset updating triggering condition, updating the corresponding nodes and edges in the knowledge graph according to a message transmission mechanism of the graph neural network; Generating natural language description according to a preset prompting word template and the multi-modal data; performing semantic speculation on the natural language description by adopting a large language model to generate a new triplet and a corresponding confidence coefficient, wherein the triplet is in an entity-relation-entity format; and adding a new triplet with the confidence coefficient larger than a preset confidence coefficient threshold value into the knowledge graph, and updating the knowledge graph. As an improvement of the above solution, the fusing the data features and the semantic features to generate feature vectors combining quantized data and semantic knowledge, denoted as target featu