CN-117274121-B - Remote sensing image fusion method of four-branch neural network and computer readable medium

CN117274121BCN 117274121 BCN117274121 BCN 117274121BCN-117274121-B

Abstract

The invention provides a four-branch neural network remote sensing image fusion method and a computer readable medium. The method comprises the steps of carrying out spectrum dimension downsampling processing on each original hyperspectral image to obtain each multispectral image with low spectrum resolution, carrying out space dimension blurring processing and downsampling processing on each original hyperspectral image to obtain each hyperspectral image with low space resolution, constructing a four-branch neural network model, carrying out iterative optimization on each hyperspectral image with low spectrum resolution and each hyperspectral image with low space resolution to obtain a trained four-branch neural network model, collecting the hyperspectral images in real time, and fusing the hyperspectral images collected in real time and the trained four-branch neural network model to obtain each predicted real-time hyperspectral image. The invention can fully extract the spatial features and the spectral features from hyperspectral and multispectral images, and has great advantages in terms of computational complexity and spatial complexity.

Inventors

YANG GUANGYI
WANG MENGYI
WANG WENGAO
HUANG HE
CAO WEINAN
ZHANG HONGYAN
HE WEI
JIN WEIZHENG

Assignees

武汉大学

Dates

Publication Date: 20260505
Application Date: 20230822

Claims (5)

1. A four-branch neural network remote sensing image fusion method is characterized in that: Performing downsampling processing of spectrum dimensions on each original hyperspectral image to obtain multispectral images with each low spectrum resolution; Carrying out fuzzy processing and downsampling processing of space dimensions on each original hyperspectral image to obtain each hyperspectral image with low space resolution; Sequentially cascading a feature extraction module, a feature fusion module and an image recovery module to construct a four-branch neural network model, and combining each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution to obtain a trained four-branch neural network model through iterative optimization of an ADAM optimizer; the four-branch neural network model comprises a feature extraction module, a feature fusion module and an image recovery module, wherein the feature extraction module, the feature fusion module and the image recovery module are sequentially cascaded, the feature extraction module is formed by connecting a first feature extraction sub-module, a second feature extraction sub-module, a third feature extraction sub-module and a fourth feature extraction sub-module in parallel, the feature fusion module comprises a first feature fusion sub-module, a second feature fusion sub-module and a third feature fusion sub-module, the first feature extraction sub-module, the second feature extraction sub-module and the first feature fusion sub-module are connected, the third feature extraction sub-module, the fourth feature extraction sub-module and the second feature fusion sub-module are connected, the third feature fusion sub-module is connected with a multi-scale module, and the image recovery module comprises a multi-scale sub-module and a multi-level sub-module which are connected; The first feature extraction submodule takes each multispectral image with low spectral resolution as input, performs feature extraction on each multispectral image with low spectral resolution to obtain the space feature of each multispectral image with low spectral resolution, the second feature extraction submodule takes each multispectral image with low spectral resolution as input, performs feature extraction on each multispectral image with low spectral resolution to obtain the spectrum feature of each multispectral image with low spectral resolution, the third feature extraction submodule takes each multispectral image with low spatial resolution as input, performs feature extraction on each multispectral image with low spatial resolution to obtain the space feature of each multispectral image with low spatial resolution, and the fourth feature extraction submodule takes each multispectral image with low spatial resolution as input, performs feature extraction on each multispectral image with low spatial resolution to obtain the spectrum feature of each multispectral image with low spatial resolution; The first feature fusion sub-module takes the space features of each multispectral image with low spectral resolution and the spectral features of each multispectral image with low spectral resolution as inputs, performs feature fusion to obtain the space spectral features of each multispectral image with low spectral resolution, and outputs the space spectral features to the third feature fusion sub-module; the second feature fusion submodule takes the spatial features of each hyperspectral image with low spatial resolution and the spectral features of each hyperspectral image with low spatial resolution as inputs, performs feature fusion to obtain the spatial features of each hyperspectral image with low spatial resolution and outputs the spatial features of each hyperspectral image with low spatial resolution to the third feature fusion submodule, and the third feature fusion submodule takes the spatial features of each multispectral image with low spatial resolution and the spatial features of each hyperspectral image with low spatial resolution as inputs, performs feature fusion to obtain the spatial features of each prediction image and outputs the spatial features of each prediction image to the multiscale submodule, wherein the first feature fusion submodule comprises a channel connection module and a 1X 1 convolution layer, the channel connection module is connected with the 1X 1 convolution layer, and the second feature fusion submodule comprises a channel connection module and a 1X 1 convolution layer, and the channel connection module is connected with the 1X 1 convolution layer; The multi-scale sub-module takes each predicted image empty spectrum feature as input to perform feature extraction to obtain multi-scale empty spectrum features of each predicted hyperspectral image, the multi-scale sub-module comprises a first empty feature extraction module, a second empty feature extraction module, a third empty feature extraction module, a fourth empty feature extraction module, a channel connection module and a 1X 1 convolution layer which are connected in parallel, the first empty feature extraction module, the second empty feature extraction module, the third empty feature extraction module and the fourth empty feature extraction module respectively take each predicted image empty spectrum feature as input to perform feature extraction to respectively obtain a first multi-scale empty spectrum feature of each predicted hyperspectral image, a second multi-scale empty spectrum feature of each predicted hyperspectral image, a third multi-scale empty spectrum feature of each predicted hyperspectral image and a fourth multi-scale empty spectrum feature of each predicted hyperspectral image, the first empty feature extraction module, the second empty feature extraction module, the third empty feature extraction module and the fourth empty feature extraction module respectively comprise a cascade connection and a first empty feature 57 PReLU, a third empty feature extraction module and a third empty feature extraction module, wherein the first empty feature extraction module and the third empty feature extraction module are connected with the third empty feature extraction module and the third empty feature extraction module respectively The multi-level sub-module takes multi-scale empty spectrum characteristics of each predicted image as input to perform characteristic extraction to obtain each predicted hyperspectral image, wherein the multi-level sub-module comprises a multi-level 1X 1 convolution layer and a multi-level channel connection module which are cascaded; And acquiring each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution in real time, and fusing the multispectral images with the low spatial resolution through a trained four-branch neural network model to obtain each predicted real-time hyperspectral image.
2. The method for fusing the remote sensing images of the four-branch neural network according to claim 1, comprising the following steps: Step1, introducing a plurality of original hyperspectral images, and performing downsampling treatment of spectrum dimensions on each original hyperspectral image to obtain multispectral images with each low spectrum resolution; Step 2, carrying out fuzzy processing and downsampling processing of space dimension on each original hyperspectral image to obtain each hyperspectral image with low space resolution; Step 3, constructing a four-branch neural network model by sequentially cascading a feature extraction module, a feature fusion module and an image restoration module, inputting each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution into the four-branch neural network model for fusion to obtain each predicted hyperspectral image, constructing a network loss function by combining each original hyperspectral image, and obtaining a trained four-branch neural network model by iterative optimization of an ADAM optimizer; And 4, collecting each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution in real time, and fusing the multispectral images through a trained four-branch neural network model to obtain each predicted real-time hyperspectral image.
3. The method for fusing remote sensing images of a four-branch neural network according to claim 1, wherein the method comprises the following steps: T feature extraction submodule comprising the first cascade connection Multispectral image space feature extraction module, the first Multispectral image space feature extraction module, the first Multispectral image space feature extraction module, the first The multispectral image space feature extraction module is t epsilon [1,4]; Said first Multispectral image space feature extraction module, the first Multispectral image space feature extraction module, the first Multispectral image space feature extraction module, the first The multispectral image space feature extraction modules comprise a depth separable convolution layer and a PReLU activation layer which are sequentially cascaded; Said first The multispectral image space feature extraction module takes each multispectral image with low spectral resolution as input, and performs space feature extraction to obtain the multispectral image with low spectral resolution Spatial features, the first The multispectral image space feature extraction module uses each multispectral image with low spectral resolution to extract The spatial feature is input, spatial feature extraction is carried out, and multispectral image of each low spectral resolution is obtained Spatial features, the first The multispectral image space feature extraction module uses each multispectral image with low spectral resolution to extract The spatial feature is input, spatial feature extraction is carried out, and multispectral image of each low spectral resolution is obtained Spatial features, the first The multispectral image space feature extraction module uses each multispectral image with low spectral resolution to extract The spatial features are input, and spatial feature extraction is carried out: If t=1, obtaining the space characteristics of each multispectral image with low spectral resolution; if t=2, obtaining the spectral characteristics of each multispectral image with low spectral resolution; if t=3, obtaining the space characteristics of each hyperspectral image with low space resolution; If t=4, obtaining the spectrum characteristics of each hyperspectral image with low spatial resolution; each multispectral image space feature with low spectral resolution and each multispectral image spectral feature with low spectral resolution are output to the first feature fusion submodule; And the high spectrum image space characteristics with each low spatial resolution and the high spectrum image spectrum characteristics with each low spatial resolution are output to the second characteristic fusion submodule.
4. The four-branch neural network remote sensing image fusion method according to claim 2, wherein the method comprises the following steps of: The network loss function model in the step3 is as follows: Wherein K represents the number of images, H represents the width of the images, W represents the height of the images and the band index, S represents the number of bands, gk (i, j, S) represents the pixel value of the ith row and jth column in the ith band in the kth original hyperspectral image, Representing the pixel value of the ith row and the jth column in the ith wave band in the kth predicted hyperspectral image.
5. A computer readable medium, characterized in that it stores a computer program for execution by an electronic device, which computer program, when run on the electronic device, causes the electronic device to perform the steps of the method according to any of claims 1-4.

Description

Remote sensing image fusion method of four-branch neural network and computer readable medium Technical Field The invention belongs to the field of remote sensing image processing, and particularly relates to a four-branch neural network remote sensing image fusion method and a computer readable medium. Background Multispectral imaging (Multi-SPECTRAL IMAGE, MSI) typically involves several bands in the visible and near infrared wavelength ranges, such as RGB color imaging. Because of the tradeoff between spectral and spatial resolution, multispectral images typically have lower spectral and higher spatial resolutions. Corresponding to this is a hyperspectral image (Hyper-SPECTRAL IMAGE, HSI), which typically contains 100 to 200 spectral bands, with higher spectral resolution and lower spatial resolution. Therefore, the fusion of the two images to obtain the dual-high image with high spectral resolution and spatial resolution has strong practical significance. Currently, the fusion method of hyperspectral and multispectral images can be roughly divided into two types, namely a traditional model-based method and a deep learning method based on a network. Because the fusion problem can be regarded as an inverse underdetermined problem, the conventional model-based method generally performs prior assumption on the dual-high image to serve as a regularization term, and common prior includes sparse prior, gaussian prior and low-rank prior. Although the method has a solid mathematical basis, the linear observation model applied by the method is difficult to fully describe the nonlinear relation between data, so that the fusion effect is affected. The deep learning (DEEP LEARNING, DL) method can fully describe the nonlinear relation between data by benefiting from strong feature extraction capability, and common network structures are convolutional neural networks, residual error networks and three-dimensional convolutional neural networks. The convolution kernel in the convolution neural network can only move in the spatial direction, so that the spatial spectrum information of the remote sensing image cannot be fully extracted. The three-dimensional convolutional neural network can solve the problem, and the convolutional kernel can slide in three dimensions of space and spectrum and can extract spatial spectrum information. However, the output of the three-dimensional convolution network can cause huge space storage requirements and computational complexity, and bring huge performance pressure to the computer. In summary, how to design a feature extraction module which can fully extract the spatial spectrum features and has low computational complexity and space complexity has great significance. Disclosure of Invention In view of the above problems, the present invention provides a four-branch neural network remote sensing image fusion method and a computer readable medium. Traditional two-dimensional convolutional neural networks focus on spatial information too much, while spectral information is ignored. Three-dimensional convolutional neural networks solve this problem with convolutional kernels that can slide in three dimensions, but their high computational complexity and spatial complexity bring tremendous performance pressure to computers. The technical scheme of the method is a four-branch neural network remote sensing image fusion method, which is characterized in that: Performing downsampling processing of spectrum dimensions on each original hyperspectral image to obtain multispectral images with each low spectrum resolution; Carrying out fuzzy processing and downsampling processing of space dimensions on each original hyperspectral image to obtain each hyperspectral image with low space resolution; Sequentially cascading a feature extraction module, a feature fusion module and an image recovery module to construct a four-branch neural network model, and combining each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution to obtain a trained four-branch neural network model through iterative optimization of an ADAM optimizer; And acquiring each multispectral image with low spectral resolution and each hyperspectral image with low spatial resolution in real time, and fusing the multispectral images with the low spatial resolution through a trained four-branch neural network model to obtain each predicted real-time hyperspectral image. The method comprises the following specific steps: Step1, introducing a plurality of original hyperspectral images, and performing downsampling treatment of spectrum dimensions on each original hyperspectral image to obtain multispectral images with each low spectrum resolution; Step 2, carrying out fuzzy processing and downsampling processing of space dimension on each original hyperspectral image to obtain each hyperspectral image with low space resolution; Step 3, constructing a four-branch neural network model, inp