Search

CN-121982563-A - Zero-sample hyperspectral remote sensing image classification method based on knowledge graph semantic representation

CN121982563ACN 121982563 ACN121982563 ACN 121982563ACN-121982563-A

Abstract

The invention relates to the technical field of intelligent processing and analysis of remote sensing images, and discloses a zero-sample hyperspectral remote sensing image classification method based on semantic representation of a knowledge graph. And simultaneously, the visual characteristics of the hyperspectral image are extracted by using a convolutional neural network. Furthermore, cross-modal distribution alignment of visual features and semantic features in the shared hidden layer space is performed by a dual-variational self-encoder. And finally, taking hidden semantic features corresponding to the unknown categories in the knowledge graph as category prototypes, generating enhanced features and training an unknown category classifier, so as to realize effective identification of the unknown ground object categories under the condition that only known category labeling samples exist. The method and the device remarkably reduce the dependence on a large amount of annotation data, and improve the classification precision and generalization capability under the zero sample scene.

Inventors

  • TANG RUIJIE
  • MA LI

Assignees

  • 中国地质大学（武汉）

Dates

Publication Date
20260505
Application Date
20260116

Claims (10)

  1. 1. A zero sample hyperspectral remote sensing image classification method based on knowledge graph semantic representation is characterized by comprising the following steps: s1, acquiring hyperspectral remote sensing image data, and preprocessing the image data; S2, constructing a domain knowledge graph containing ground object category entities, attribute entities and relationships among the entities based on prior knowledge of the remote sensing domain; S3, constructing and training a semantic feature extraction module, namely adopting a knowledge graph to represent a learning model, taking entities and relation data in the domain knowledge graph constructed in the S2 as training samples to train the model, and obtaining an optimized semantic feature extraction module which can output low-dimensional dense semantic feature representation corresponding to any feature type in the knowledge graph; s4, constructing a zero sample classification model, wherein the model comprises a semantic feature extraction module, a visual-semantic cross-modal alignment module and an unknown class classification module, and further comprises three modules with learnable parameters in a training stage; S5, training the visual feature extraction module, namely training a visual feature extraction network by using a known class labeling sample in the hyperspectral image as input to obtain an optimized visual feature extraction module; S6, training the visual-semantic cross-modal alignment module, namely inputting the known category visual features extracted in the S5 and the corresponding known category semantic features extracted in the S3 simultaneously, and performing cross-modal distribution alignment and reconstruction training in a shared hidden layer space through a double variation self-encoder to obtain an optimized visual-semantic cross-modal alignment module; S7, training the unknown class classification module, namely processing the unknown class semantic features extracted in the S3 by utilizing a semantic encoder in the S6 to obtain unknown class hidden layer semantic features in an S6 shared hidden layer space, and taking the unknown class hidden layer semantic features as class prototypes; and S8, model testing, namely inputting an unknown class sample to be classified in the hyperspectral image into the trained zero sample classification model, sequentially passing through the visual feature extraction module and the visual encoder in the visual-semantic cross-modal alignment module, and finally outputting the unknown class label to which the unknown class classification module belongs.
  2. 2. The method for classifying the hyperspectral remote sensing images of the zero sample based on the semantic representation of the knowledge graph of claim 1, wherein the step S2 specifically comprises the following steps: s21, taking the ground object categories appearing in the hyperspectral image dataset as core entities; S22, defining a plurality of relationship types including a hierarchical relationship, an attribute relationship and a spatial relationship; S23, constructing a knowledge graph expressed in a fact triplet form based on the entity and the relation.
  3. 3. The method for classifying a hyperspectral remote sensing image with zero sample based on semantic representation of a knowledge graph according to claim 1, wherein in the step S3, the learning model of the semantic representation of the knowledge graph is DistMult model, which is trained by minimizing the loss based on a triplet scoring function, so as to map the entities and the relations in the knowledge graph into semantic vectors.
  4. 4. The method for classifying the hyperspectral remote sensing images of the zero sample based on the semantic representation of the knowledge graph of claim 1, wherein in the step S5, the visual feature extraction network is a two-dimensional convolutional neural network, and only labeling samples of known categories are used in the training process.
  5. 5. The method for classifying the hyperspectral remote sensing images of the zero sample based on the semantic representation of the knowledge graph of claim 1, wherein the step S6 specifically comprises the following steps: S61, coding and reconstructing the input known category visual features by using a visual variation self-coder; s62, coding and reconstructing the input semantic features of the known category by using a semantic variation self-coder; s63, cross-modal alignment is achieved by minimizing the distribution distance of visual and semantic features in the hidden layer space.
  6. 6. The method for classifying the hyperspectral remote sensing images of the zero sample based on the semantic representation of the knowledge-graph according to claim 5, wherein the distribution distance in the S63 is minimized by a loss function of reconstruction loss, cross-modal feature reconstruction loss and cross-modal feature alignment loss of the visual and semantic variation self-encoder.
  7. 7. The method for classifying the hyperspectral remote sensing images of the zero sample based on the semantic representation of the knowledge graph of claim 1, wherein the step S7 specifically comprises the following steps: s71, extracting semantic vectors of unknown category entities from the semantic feature representation obtained in the S3; s72, mapping the unknown category semantic vector into the shared hidden layer space obtained in the S7 by using a semantic encoder, and taking the unknown category semantic vector as a category prototype; S73, repeating and randomly perturbing the class prototype to generate an unknown class hidden layer representation sample with enhanced data; S74, training a classifier consisting of a linear layer and a Softmax function by using the unknown class hidden layer representation sample after data enhancement.
  8. 8. The method for classifying the hyperspectral remote sensing images of zero samples based on semantic representation of knowledge graph according to claim 1, wherein the overall optimization objective function L adopted in the training process of S5 to S7 is defined as L=L CE + L KG + L UNK , wherein L CE is cross entropy classification loss of a visual feature extraction module, L KG is loss of a visual-semantic cross-modal alignment module, and L UNK is loss of an unknown class classifier.
  9. 9. A storage medium is characterized in that the storage medium stores instructions and data for realizing the zero-sample hyperspectral remote sensing image classification method based on semantic representation of a knowledge graph according to any one of claims 1-8.
  10. 10. The zero-sample hyperspectral remote sensing image classification device based on the knowledge graph semantic representation is characterized by comprising a processor and a storage medium, wherein the processor loads and executes instructions and data in the storage medium to realize the zero-sample hyperspectral remote sensing image classification method based on the knowledge graph semantic representation according to any one of claims 1-8.

Description

Zero-sample hyperspectral remote sensing image classification method based on knowledge graph semantic representation Technical Field The invention relates to the technical field of intelligent processing and analysis of remote sensing images, in particular to a zero-sample hyperspectral remote sensing image classification method based on semantic representation of a knowledge graph. Background The hyperspectral image (HYPERSPECTRAL IMAGE, HIS) is a three-dimensional stereoscopic image captured by an aerospace vehicle carrying a hyperspectral imager, and each pixel in the image contains hundreds of spectral reflection information in different wavebands. The method has obvious advantages in the fields of ground object classification, environment monitoring, resource investigation and the like. The hyperspectral remote sensing image ground object classification is one of the research emphasis and research hotspots in the hyperspectral remote sensing field, and each pixel point in the image is distributed into different ground object categories by utilizing multiband data acquired by a hyperspectral sensor. In recent years, with the development of imaging technology and the improvement of data acquisition capability, the scale of hyperspectral remote sensing data is continuously enlarged, and the wide application of a deep learning model in the field is promoted. Under the condition of fully labeling the ground objects, the method can generally obtain higher classification precision. However, the method is highly dependent on the labeling sample, the hyperspectral image pixel-level labeling needs a large amount of manual participation, the labeling cost is high, the period is long, and the method is difficult to adapt to the rapid updating requirement of large-scale remote sensing data. In practical remote sensing application, the remote sensing image contains complicated ground object types, the manual investigation range is limited, labeling of all ground objects is difficult to ensure, and unlabeled ground object types may exist in the image. Under the background, the traditional supervised learning method depending on complete annotation information is difficult to effectively identify unlabeled categories in the data set, and the unlabeled categories are wrongly classified into annotated categories, so that the image classification accuracy is affected. In order to alleviate the incomplete problem of category labeling, the zero sample learning technology only utilizes labeling samples of known categories in a training stage by introducing auxiliary information such as category semantics, so as to realize the identification of unlabeled target categories, namely unknown categories. However, the existing zero-sample hyperspectral classification method generally adopts manually constructed attribute vectors or universal natural language word vectors as category semantic representations, the category semantic information is loose in structure and weak in field pertinence, hierarchical relationships, spatial correlations and spectrum discrimination attributes among hyperspectral remote sensing features are difficult to characterize, obvious offset exists between semantic space and visual space, and therefore unknown category recognition performance is restricted. In addition, the partial zero sample method only adopts an embedded alignment strategy, performs simple mapping in a feature space, and is difficult to explicitly model class condition distribution, and the generated zero sample method has certain advantages, but is difficult to fully exert potential under the condition of insufficient semantic information expression capability. Therefore, how to construct a semantic representation which is structured, has strong discriminant and is compatible with the application characteristics of hyperspectral remote sensing, and effectively introduce the semantic representation into a generated zero sample classification framework is a problem which needs to be solved in the prior art. Disclosure of Invention The invention aims to provide a zero-sample hyperspectral remote sensing image classification method based on knowledge graph semantic representation, which solves the technical problems that the prior art cannot construct semantic representation which is structured, has strong discrimination and is matched with hyperspectral remote sensing application characteristics, and the semantic representation is effectively introduced into a generated zero-sample classification frame. Specifically, according to the zero-sample hyperspectral remote sensing image classification method based on the semantic representation of the knowledge graph, the knowledge graph in the remote sensing field is constructed, and the zero-sample learning framework is generated in a combined mode, so that the cross-modal alignment of hyperspectral visual features and structural semantic features is realized, and the unknown class identific