CN-121998876-A - Automatic fisheye image distortion correction method based on global and local feature fusion

CN121998876ACN 121998876 ACN121998876 ACN 121998876ACN-121998876-A

Abstract

The invention discloses an automatic fisheye image distortion correction method based on global and local feature fusion, which is used for constructing an automatic fisheye image distortion correction model based on global and local feature fusion, wherein the model comprises a global feature extraction module, a local feature extraction module, an attention mechanism weighting fusion module, a distortion parameter estimation module and a distortion correction module. Firstly, an automatic distortion correction data set of the fisheye image is constructed, then the data set is utilized to train the proposed model, the trained model is used for testing the input fisheye image, and distortion parameters and distortion correction images are generated. The model provided by the invention carries out weighted fusion feature extraction on the global features and the local features of the fisheye image, so that the model can focus on the linear semantic features beneficial to distortion correction, the accuracy of estimating the distortion parameters of the fisheye image is improved, the distortion correction effect of the fisheye image is improved, and the model has wide application prospect in the field of distortion correction of the fisheye image.

Inventors

CAO JIHAO
LI WEILI
DENG JINSHENG
YIN XIAOJING
LI WUYANG
REN TIANXIANG

Assignees

中国人民解放军国防科技大学

Dates

Publication Date: 20260508
Application Date: 20251218

Claims (8)

1. The automatic fish-eye image distortion correction method based on global and local feature fusion mainly comprises the following steps: Constructing an automatic fisheye image distortion correction simulation data set, wherein each sample in the automatic fisheye image distortion correction simulation data set comprises a fisheye image, a corresponding semantic segmentation result and distortion parameters; step two, constructing an automatic fisheye image distortion correction model based on global and local feature fusion, wherein the model comprises a global feature extraction module, a local area information extraction module, an attention mechanism weighting fusion module, a distortion parameter estimation module and a distortion correction module; step three, iterative training is carried out on the automatic fisheye image distortion correction model by utilizing the data set constructed in the step one, so as to obtain the trained automatic fisheye image distortion correction model based on global and local feature fusion; step four, utilizing a trained fisheye image automatic distortion correction model based on global and local feature fusion to extract fusion features and estimate distortion parameters of a tested fisheye image, and finally generating a distortion correction image; The global feature extraction module aims at extracting global features of an input fisheye image, overall grasps distortion degree and distortion parameter information of the whole input fisheye image, the local area information extraction module performs semantic segmentation on the input fisheye image, selects a specific area with rich linear information in a semantic segmentation result to encode, obtains local area features, connects the global feature extraction module and the local area information extraction module with the attention mechanism weighting fusion module, enables the attention mechanism weighting fusion module to pay attention to features in the specific area, generates a fisheye distortion parameter result by using the distortion parameter estimation module, and obtains a final fisheye distortion correction image by using the distortion correction module.
2. The method for automatically correcting distortion of fish-eye image based on fusion of global and local features as set forth in claim 1, wherein the global feature extraction module uses the output of the middle convolution layer of the pre-trained deep neural network as global feature representation, and the sign is that Wherein The local area information extraction module inputs fisheye images through a semantic segmentation network Performing pixel-level class prediction to obtain a semantic mask Screening out specific areas rich in linear information according to semantic masks, performing feature coding on the screened specific areas, and extracting local features Design of a global feature combining spatial and channel attention modules And local features And carrying out weighted fusion, wherein a weighted fusion formula is defined as follows: ; Wherein, the In order to fuse the features of the features, Is a weight parameter which can be learned, meets the following conditions ; For the attention weight graph, the following is calculated: ; Wherein, the Representing the Sigmoid activation function, Representing an element-by-element multiplication, Representing a combined spatial and channel attention module operation; The distortion parameter estimation module is used for fusing characteristics Mapping to fish-eye distortion parameter space through a series of convolution layers and full connection layers to output distortion parameter vector Wherein Represent the first And (3) setting distortion model parameters as a simplified radial distortion model, wherein a sampling formula is as follows: ; Wherein, the For the angle between the viewing angle and the optical axis, The pixel radius after corresponding distortion; Using estimated distortion parameters in a distortion correction module Correcting the input fisheye image by adopting reverse distortion mapping, and recovering an undistorted image 。
3. The method for automatically correcting distortion of fish-eye image based on global and local feature fusion according to claim 2, characterized by an attention mechanism Designed to incorporate channel attention And spatial attention Is the product of: ; Wherein, the Representing global average pooling function, output size , Weight matrix in channel attention, A convolution kernel parameter representing spatial attention, Respectively representing convolution, matrix multiplication, element-by-element product.
4. The method for automatically correcting distortion of fish-eye image based on global and local feature fusion according to claim 2, wherein the weight parameters in the model Iterative updating is carried out in a soft constraint form, and learning is guided through the following regularization loss: ; Wherein, the In order to balance the regular term weight, the stability and sparsity of the fusion weight are ensured.
5. The method for automatically correcting distortion of a fisheye image based on global and local feature fusion of claim 2 wherein a dynamic selection mechanism of a local region is designed in the local region information extraction module and based on a local region straightness evaluation function Metric area The line segment characteristics of the inner edge, , wherein, Is a region All the detected edge line segments are collected in the interior, Evaluating the linear degree of a single line segment e, and only selecting the line segment which meets the threshold value Is feature-coded to ensure the validity of the local feature.
6. The method for automatically correcting the distortion of the fisheye image based on the fusion of global and local features as set forth in claim 2, wherein the model for automatically correcting the distortion of the fisheye image includes regression loss by combining distortion parameters And image re-projection error loss Defining a joint loss function: ; Wherein the distortion parameter is regressed and lost , In order to predict the parameters of the model, For true annotation parameters, the image re-projection error loss is calculated by comparing the corrected image keypoints with the true points before distortion, 、 And Is a weight parameter.
7. The method for automatically correcting distortion of a fisheye image based on global and local feature fusion according to claim 1, wherein the existing semantic segmentation dataset in the first step comprises an ADE20K dataset, the distortion parameters of the fisheye image comprise center coordinates of a fisheye region and distortion polynomial coefficients, and the sample number dividing ratio of a training set, a verification set and a test set is 6:2:2.
8. The method for automatically correcting the distortion of the fisheye image based on the fusion of global and local features according to claim 1, wherein the training process of the model proposed in the third step is completed by using PyTorch, tensorflow or MXNet deep learning frames.

Description

Automatic fisheye image distortion correction method based on global and local feature fusion Technical Field The invention relates to an automatic distortion correction technology for a fisheye image, in particular to an automatic distortion correction method for the fisheye image based on global and local feature fusion. Background The fish-eye image distortion correction technology aims at estimating distortion parameters from an input fish-eye image, transforming the fish-eye image based on the distortion parameter estimation result, removing distortion in the fish-eye image, and obtaining a distortion corrected image. In the actual fisheye image distortion correction process, the image region corresponding to the semantic category (such as buildings, roads and the like) with rich linear characteristics often contains more structural deformation information than the region corresponding to other categories, so that the method has more significance for the fisheye image distortion parameter estimation and distortion correction process. The traditional fisheye image distortion correction method mainly focuses on long circular arc characteristics in the fisheye image, is limited by the extraction precision of a long circular arc object, has low iterative solving process speed, and has unstable distortion correction effect. In addition, the traditional method does not pay attention to specific semantic categories with rich straight line characteristics, and the fish-eye distortion parameter estimation accuracy cannot be further improved by utilizing the information. Aiming at the problems, the patent provides an automatic fisheye image distortion correction method based on global and local feature fusion, adopts a deep neural network model to fully mine global and local features of the fisheye image, carries out weighted fusion on the features, and further improves the fisheye distortion parameter estimation precision and the distortion correction effect on the basis. Disclosure of Invention In view of the foregoing, it is necessary to provide an automatic fisheye image distortion correction method based on global and local feature fusion. The method constructs an automatic fisheye image distortion correction simulation data set, proposes an automatic fisheye image distortion correction model based on fusion of global and local features, and trains the proposed model by utilizing the data set. And testing the input fisheye image by the trained model to generate distortion parameters and distortion correction images. The automatic fish-eye image distortion correction method based on global and local feature fusion mainly comprises the following steps: Step one, constructing a fisheye image automatic distortion correction simulation data set. Based on the existing semantic segmentation data set, setting fish-eye distortion parameters, and transforming normal undistorted images and semantic segmentation results in the semantic segmentation data set according to a fish-eye distortion model to generate simulated fish-eye images and corresponding semantic segmentation results. Each sample in the fisheye image automatic distortion correction simulation dataset includes a fisheye image, a corresponding semantic segmentation result, and a distortion parameter. The data set is divided into a training set, a verification set and a test set. Existing semantic segmentation datasets include ADE20K datasets and the like. The distortion parameters of the fisheye image comprise the center coordinates of the fisheye region and distortion polynomial coefficients. The sample number dividing ratio of the training set, the verification set and the test set is 6:2:2. And secondly, constructing an automatic fisheye image distortion correction model based on global and local feature fusion. The model mainly comprises a global feature extraction module, a local area information extraction module, an attention mechanism weighting fusion module, a distortion parameter estimation module and a distortion correction module. The global feature extraction module aims at extracting global features of the input fisheye image and grasping distortion degree and distortion parameter information of the whole input image as a whole. The module can be realized through a pre-trained deep neural network VGG16, and the output result of the middle convolution layer is selected as a global feature. The local region information extraction module performs semantic segmentation on the input fisheye image, selects a specific region with rich linear information in a semantic segmentation result to encode, obtains local region characteristics, and keeps the width and the height of the characteristic dimension consistent with the global characteristics. The specific semantic areas rich in straight line features comprise areas such as buildings and roads. The global feature extraction module and the local region information extraction module are connected with