CN-121998059-A - Dynamic situation updating and acquiring method based on multi-mode knowledge graph

CN121998059ACN 121998059 ACN121998059 ACN 121998059ACN-121998059-A

Abstract

The invention discloses a dynamic situation updating and acquiring method based on a multi-modal knowledge graph, which belongs to the technical field of natural language processing and comprises the steps of collecting multi-source multi-modal data, preprocessing the multi-source multi-modal data to generate standardized data with a timestamp and a position tag, defining a body, realizing cross-modal feature alignment and semantic mapping, generating a multi-modal knowledge triplet, constructing the multi-modal knowledge graph, realizing automatic incremental updating and timing full-quantity verification of the multi-modal knowledge graph by combining a data increment threshold value and a time period, and outputting a high-credibility updating result based on data credibility weighting and multi-source modal verification to realize minute-level updating of the situation. The invention solves the problems of weak fusion capability, insufficient dynamic update efficiency and accuracy and uncertainty data processing deficiency of the existing multi-mode data. The method can improve the utilization rate of multi-mode data, reduce situation update delay, improve situation credibility and enhance situation interpretability.

Inventors

CHEN JIE
TAO YE
TANG ZILU

Assignees

合肥行至智能科技有限公司

Dates

Publication Date: 20260508
Application Date: 20260202

Claims (8)

1. The dynamic situation updating and acquiring method based on the multi-mode knowledge graph is characterized by comprising the following steps of: Synchronously acquiring image, signal, text and voice data based on a standardized interface, determining multi-source multi-mode data, and embedding a time stamp and a position tag in the acquisition process to trace each piece of data to a specific space-time dimension; preprocessing multi-source multi-mode data based on different mode data characteristics, and generating standardized data with a time stamp and a position tag through target feature extraction, signal noise reduction, entity identification and voice-text conversion operation; Defining an ontology containing combat entities, relations and attributes, realizing cross-modal feature alignment through an attention mechanism, establishing semantic mapping based on a multi-modal dictionary, generating a multi-modal knowledge triplet with a credibility label, and constructing a multi-modal knowledge graph; And (3) combining the data increment threshold value and the time period to realize automatic increment updating and timing full-quantity verification of the multi-mode knowledge graph, outputting a high-reliability updating result based on data reliability weighting and multi-source mode verification, marking low-reliability data to be confirmed and feeding back, and realizing minute-level updating of the situation.
2. The method for dynamically updating and acquiring situation based on multi-modal knowledge graph according to claim 1, wherein the method for synchronously acquiring image, signal, text and voice data based on the standardized interface, determining multi-source multi-modal data, and performing the following operations: Acquiring image data based on satellites, unmanned aerial vehicles and ground cameras, wherein the image data cover visual information of topography, equipment appearance and personnel deployment; collecting signal data based on radar and electronic reconnaissance equipment, wherein the signal data comprises signal information of target speed, azimuth and electromagnetic characteristics; Text data is collected based on a command system and an information terminal, and the text data comprises structured or unstructured texts of combat instructions, information reports and equipment parameters; Collecting voice data based on soldier interphones and command radio stations, wherein the voice data comprise voice information for real-time tactical communication and emergency feedback; and forming multi-source and multi-mode data according to the acquired image data, signal data, text data and voice data.
3. The method for dynamically updating and acquiring situation based on multi-modal knowledge graph according to claim 2, wherein the following operations are performed by preprocessing multi-source multi-modal data based on different modal data characteristics: preprocessing image data based on a convolutional neural network, extracting target features, generating feature vectors, preprocessing the image data based on an image enhancement algorithm, and reducing the influence of smoke and dust on image quality; Preprocessing signal data based on wavelet transformation, filtering radar clutter, extracting target motion characteristics including speed and acceleration, and performing spectrum analysis on electromagnetic signals to generate signal fingerprints; preprocessing text data based on a BERT model, realizing text depth semantic representation through bi-directional context understanding, performing word segmentation and entity recognition, and converting unstructured text into structured triples; And preprocessing voice data based on an automatic voice recognition technology, converting voice into text, and executing a text preprocessing flow, so as to reserve voice emotion characteristics and assist in judging the emergency degree.
4. The method for updating and acquiring dynamic situation based on multi-modal knowledge graph according to claim 3, wherein the method is characterized in that an ontology containing combat entities, relations and attributes is defined, cross-modal feature alignment is realized through an attention mechanism, semantic mapping is established based on a multi-modal dictionary, multi-modal knowledge triples with credibility labels are generated, the multi-modal knowledge graph is constructed, and the following operations are executed: Performing body structure definition, designing a special body, and comprising three core elements of an entity, a relationship and an attribute, wherein the entity comprises a combat unit, equipment, topography and personnel, the relationship comprises membership, deployment and attack, and the attribute comprises an entity attribute, a time attribute and a credibility attribute; the method comprises the steps of performing multi-modal data fusion based on a dual-stage fusion scheme of feature alignment and semantic mapping, wherein image feature vectors, signal fingerprints and text entity features are mapped to the same semantic space through a attention mechanism, and cross-modal entity association is established based on a multi-modal dictionary generated by a pre-training large model to generate multi-modal knowledge triples; The graph database is used for storing a multi-mode knowledge graph, supporting the rapid query and modification of entities, relations and attributes, and attaching a time stamp and a credibility label to each triplet for subsequent updating and situation assessment.
5. The method for dynamically updating and acquiring a situation based on a multi-modal knowledge graph according to claim 4, wherein attaching a credibility tag to each triplet comprises: defining a data source credibility benchmark score, and dividing data source weights according to military information grades; The satellite or early warning machine data primary source is 90-100, the radar or unmanned plane data secondary source is 70-89, the manual reconnaissance or open source information tertiary source is 40-69, and the non-verification communication interception quaternary source is 0-39; If the triplet is supported by n data sources together, taking a weighted average, and if a conflict source exists, deducting the conflict penalty.
6. The method for acquiring dynamic situation update based on a multi-mode knowledge graph according to claim 5, wherein the method is characterized in that the automatic incremental update and the timed full verification of the multi-mode knowledge graph are realized by combining a data increment threshold value and a time period, a high-reliability update result is output based on data reliability weighting and multi-source mode verification, low-reliability data is marked to be confirmed and fed back, the minute-level update of the situation is realized, and the following operations are executed: setting an increment triggering mechanism, wherein the increment triggering mechanism comprises two triggering conditions, namely data increment triggering and time timing triggering; When the time is triggered at fixed time, the full-quantity situation verification is triggered according to a fixed period, so that the situation is still accurate when no incremental data exists; setting conflict detection and solution, and adopting a reliability weighting and multisource verification strategy aiming at the conflict between new data and existing information of the map; The multi-source verification is to automatically associate other mode data verification if the single data reliability is low, and mark the command end to be confirmed and fed back when the verification is impossible; Setting an update execution scheme, and executing three types of operations on the map according to conflict processing results, wherein the operations comprise adding, modifying and deleting; Wherein, the new addition is to add new entity and new relation, the modification is to update entity attribute and relation state, and the deletion is to remove failure information.
7. The method for dynamically updating and acquiring situation based on multi-modal knowledge graph according to claim 6, wherein when the data increment is triggered, when the newly acquired multi-modal data amount reaches a threshold value, the update is automatically triggered when the system is idle, and the following operations are performed: writing all accessed image, signal, text and voice data into a high-performance message queue after preliminary time stamping and information source marking, wherein the message queue is used for decoupling, buffering and flow shaping; When the data increment is triggered, the unprocessed data quantity of each information source or key entity type in the information queue is monitored, when the data quantity is accumulated to reach a preset threshold value and the system load is lower than an idle time threshold value, an increment updating flow is immediately triggered, wherein the system is judged to acquire the system CPU, the memory and the I/O load in real time through a resource monitor when the system is idle, so that an updating task does not influence an online high-priority transaction, cross-modal association is executed on newly added data, new attributes or relations of new entities or existing entities are identified, real-time conflict detection is carried out, the conflict situation of positions and states is found out by comparing the newly added data with the existing data in the map, the fused result is updated to the map in an increment transaction mode, and a map updated event is issued through an event bus after the map is updated.
8. The method for dynamically updating and acquiring situation based on multi-modal knowledge graph according to claim 7, wherein when time is triggered, full situation verification is triggered according to a fixed period, so that situation is still accurate without incremental data, and the following operations are performed: writing all accessed image, signal, text and voice data into a high-performance message queue after preliminary time stamping and information source marking, wherein the message queue is used for decoupling, buffering and flow shaping; When time is triggered, the timer independent of the data flow is triggered once every interval period, the system load is checked after the triggering, if the system is busy, the short delay is carried out or the granularity of the verification is reduced, but the verification is required to be carried out, the current whole knowledge graph and all external data source snapshots are checked, the hard constraint is checked based on rules, the logic contradiction detection is carried out based on reasoning and the integrity of the graph, the long-time unrefreshed data is attenuated according to a preset model, the reliability of a key entity is recalculated by combining multi-source information, the state of the entity which does not have new data but needs to be dynamic is speculated by combining the last known speed, the path and the task of the entity, the active detection request is possibly triggered, and the graph updated event is issued through an event bus after the graph is updated.

Description

Dynamic situation updating and acquiring method based on multi-mode knowledge graph Technical Field The invention relates to the technical field of natural language processing, in particular to a dynamic situation updating and acquiring method based on a multi-mode knowledge graph. Background The dynamic situation update is a core support for command decision, the core requirement is that multisource information is integrated accurately in real time and an interpretable situation view is generated, the current information presents typical multi-mode characteristics, image data acquired by a satellite/unmanned aerial vehicle, signal data generated by a radar/sensor, text instruction data transmitted by communication equipment, voice data fed back by a soldier terminal and the like are covered, and various data formats have large differences and weak semantic association and are difficult to directly use for situation analysis. Knowledge graph technology is used as a semantic data integration tool, is initially applied in the field of information analysis, and realizes structured knowledge expression by defining entities (such as combat units, equipment and topography), relationships (such as membership, deployment and attack) and attributes (such as position, state and warfare). However, the environment has high dynamic performance (equipment movement, state real-time change), data uncertainty (signal interference and information error) and multi-mode cooperative requirements, and the conventional static knowledge graph cannot meet the combat requirements of situation real-time update, so that a situation acquisition technology system supporting multi-mode data fusion and dynamic update needs to be constructed. The related technology of the current situation acquisition and the knowledge graph has the following core defects in three aspects, and the actual combat requirement is difficult to meet: 1) The multi-mode data fusion capability is weak, namely the prior art is used for constructing a knowledge graph aiming at single-mode data (such as processing text information or image targets only) and lacks a cross-mode association mechanism, for example, a tank cluster identified in an image cannot be used for establishing entity association with red Fang Zhuangjia travel in a text instruction, so that multi-source information fragmentation is caused, and situation integrity is insufficient. 2) The dynamic updating efficiency and accuracy are insufficient, the traditional knowledge graph updating depends on manual labeling or batch data retraining, the updating period is as long as hours or even days, the 'minute-level' situation change requirement cannot be matched, meanwhile, data conflict (such as reporting of the position deviation of the same target by different sensors) is not considered in the updating process, and situation misjudgment is easily caused. 3) Uncertainty data processing is missing, data often accompanies noise (such as radar clutter), missing (such as partial area data blank caused by communication interruption) or ambiguity (such as equipment model cannot be determined by a blurred image), situation information is incomplete due to direct filtering of the data in the prior art, or a default value is simply filled, so that situation credibility is reduced. Disclosure of Invention The invention aims to provide a dynamic situation updating and acquiring method based on a multi-mode knowledge graph, which can improve the utilization rate of multi-mode data, reduce situation updating delay, improve situation credibility and enhance situation interpretability, and solve the problems in the background technology. In order to achieve the above purpose, the present invention provides the following technical solutions: the dynamic situation updating and acquiring method based on the multi-mode knowledge graph comprises the following steps: Synchronously acquiring image, signal, text and voice data based on a standardized interface, determining multi-source multi-mode data, and embedding a time stamp and a position tag in the acquisition process to trace each piece of data to a specific space-time dimension; preprocessing multi-source multi-mode data based on different mode data characteristics, and generating standardized data with a time stamp and a position tag through target feature extraction, signal noise reduction, entity identification and voice-text conversion operation; Defining an ontology containing combat entities, relations and attributes, realizing cross-modal feature alignment through an attention mechanism, establishing semantic mapping based on a multi-modal dictionary, generating a multi-modal knowledge triplet with a credibility label, and constructing a multi-modal knowledge graph; And (3) combining the data increment threshold value and the time period to realize automatic increment updating and timing full-quantity verification of the multi-mode knowledge graph, outputting a h