CN-121986354-A - Image restoration method and device, electronic equipment and storage medium

CN121986354ACN 121986354 ACN121986354 ACN 121986354ACN-121986354-A

Abstract

The application provides an image restoration method, an image restoration device, electronic equipment and a storage medium, and relates to the technical field of image processing, wherein the method comprises the steps of obtaining an image to be restored; and processing the image to be repaired to obtain a repaired high-definition image according to an image repair model, wherein the image repair model is trained by a plurality of training stages, and the loss function of one training stage in the plurality of training stages is a loss function aiming at optimizing the structural information of the image, so that the image repair model can repair the structural information in the image to be repaired.

Inventors

WANG TINGTING

Assignees

京东方科技集团股份有限公司

Dates

Publication Date: 20260505
Application Date: 20240829

Claims (15)

A method of image restoration, wherein the method comprises: Acquiring an image to be repaired; processing the image to be repaired according to an image repair model to obtain a repaired high-definition image; The image restoration model is trained through a plurality of training phases, and a loss function of one training phase in the plurality of training phases is a loss function aiming at optimizing structural information of an image.
The image restoration method according to claim 1, wherein the image restoration model comprises a degradation removal network, a first control network and a generation network, wherein the image to be restored is processed according to the image restoration model to obtain a restored high-definition image, and the method comprises the following steps: Inputting the image to be repaired into the degradation removing network to carry out degradation removing treatment to obtain a degradation removed image, wherein the degradation removing treatment comprises at least one of deblurring treatment, noise removing treatment, scratch removing treatment, dust removing effect treatment, brightness enhancing treatment and image super-resolution; Inputting the degradation-free image into the first control network for feature extraction to obtain a first intermediate feature, wherein the first intermediate feature is used for guiding the generation network to generate a repair image based on degradation-free information in the degradation-free image; And inputting the noise image and the first intermediate feature which are randomly sampled into the generation network to generate a restored image, so as to obtain a restored high-definition image.
The image restoration method according to claim 2, wherein the image restoration model further includes a second control network and a structural information labeling network, the method further comprising: inputting the image to be repaired into the structural information marking network to carry out structural position marking to obtain a structural position marking image; Inputting the structural position marked image into the second control network for feature extraction to obtain a second intermediate feature, wherein the second intermediate feature is used for guiding the generating network to generate a repair image based on the structural information of the structural position marked image; Inputting the randomly sampled noise image and the first intermediate feature into the generation network to generate a restored image, and obtaining a restored high-definition image, wherein the method comprises the following steps: and inputting the randomly sampled noise image, the first intermediate feature and the second intermediate feature into the generation network to generate a restored image, so as to obtain a restored high-definition image.
A method of image restoration according to claim 2 or 3 wherein the image restoration model is trained in accordance with the steps of: Sequentially carrying out first-stage degradation removal network training and second-stage control network training on the image restoration model according to a first training data set by taking degradation removal information of an optimized image as a target to obtain a primarily trained image restoration model; Performing model fine adjustment at a third stage on the primarily trained image restoration model according to a second training data set by taking structural information of an optimized image as a target to obtain a trained image restoration model; Each training data set in the first training data set comprises a sample high-definition image and a degradation processing image, and each training data set in the second training data set comprises a specific style image and a sample high-definition image, wherein the specific style image represents an image consistent with the style of the image to be repaired.
The image restoration method according to claim 4, wherein the performing the first-stage degradation removal network training on the restoration model according to the first training data set with the objective of optimizing the degradation removal information of the image includes: inputting the degradation processing image into the degradation removing network for degradation removing processing to obtain a first high-definition image; And updating network parameters of the degradation removal network according to the difference between the first high-definition image and the sample high-definition image.
The image restoration method according to claim 4, wherein, in the case where the control network is the first control network, targeting to optimize the degradation information of the image, performing the second stage of control network training on the image restoration model according to the first training data set includes: Fixing network parameters of the degradation removing network, and inputting the degradation processing image into the image restoration model through the degradation removing network to carry out image restoration to obtain a second high-definition image; and updating network parameters of the first control network according to the difference between the second high-definition image and the sample high-definition image.
The image restoration method according to claim 4, wherein, in a case where the control network includes a first control network and a second control network, targeting to optimize the degradation removal information of the image, performing the second stage of control network training on the image restoration model according to the first training data set includes: Fixing network parameters of the degradation removing network, and respectively inputting the degradation processing image into the image restoration model for image restoration through the degradation removing network and the structure information marking network to obtain a third high-definition image; And updating network parameters of the first control network and the second control network according to the difference between the third high-definition image and the sample high-definition image.
The image restoration method according to claim 4, wherein the performing a third-stage model fine-tuning of the primarily trained image restoration model according to the second training data set with the aim of optimizing structural information of the image includes: Fixing the degradation removing network, and inputting the specific style image into the primarily trained image restoration model for image restoration to obtain a fourth high-definition image; and updating network parameters of the control network according to the structural information difference of the fourth high-definition image and the sample high-definition image on the brightness channel image.
The image restoration method according to claim 4, wherein the second training data set is constructed according to the steps of: Constructing a sample high-definition image set, wherein the sample high-definition image set comprises a plurality of sample high-definition images; generating a specific style image corresponding to the sample high-definition image based on a target processing mode; And taking the sample high-definition image and the specific style image as one training data in the second training data set.
The image restoration method according to claim 9, wherein generating the specific style image corresponding to the sample high definition image based on the target processing mode comprises: Inputting the sample high-definition image into a style generation type countermeasure network for processing to obtain a specific style image corresponding to the sample high-definition image; The style generation type countermeasure network is obtained by training according to a training specific style image and a training high-definition image, and a mapping relation between the training specific style image and the training high-definition image is learned, wherein the training specific style image and the training high-definition image comprise the training specific style image and the training high-definition image which are not image pairs.
The image restoration method according to claim 9, wherein generating the specific style image corresponding to the sample high definition image based on the target processing mode comprises: Inputting the sample high-definition image into a target diffusion network for processing to obtain a specific style image corresponding to the sample high-definition image; The target diffusion network is obtained by modifying network parameters of the diffusion network through a first matrix and a second matrix, wherein the first matrix is randomly initialized according to standard normal distribution, and the second matrix is initialized to 0.
The image restoration method according to any one of claims 9 to 11, wherein the style of the specific style image includes any one of a black-and-white photo style, an old photo style.
An image restoration device, wherein the device comprises: The acquisition module is used for acquiring the image to be repaired; The restoration module is used for processing the image to be restored according to the image restoration model to obtain a restored high-definition image; The image restoration model is trained through a plurality of training phases, and a loss function of one training phase in the plurality of training phases is a loss function aiming at optimizing structural information of an image.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the image restoration method of any of claims 1-12 when the computer program is executed by the processor.
A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the image restoration method of any of claims 1-12.

Description

Image restoration method and device, electronic equipment and storage medium Technical Field The present application relates to the field of image processing technologies, and in particular, to an image restoration method, an image restoration device, an electronic device, and a storage medium. Background Image restoration techniques are becoming more and more widely used in the field of image processing, and image restoration (Image Restoration, IR) aims to improve subjective quality of images distorted by various forms of degradation, and image restoration tasks include Super Resolution (SR) of images, deblurring, denoising, compression artifact removal processing, and the like. The prior knowledge about the real-world high-definition image contained in the diffusion model plays an important role in processing image restoration, and the existing image restoration method based on the diffusion model has good restoration effects in the aspects of denoising processing, deblurring effects and the like. However, the existing image restoration method based on the diffusion model can generate other non-existing details in the restored image process, and can not well restore structural defects such as block noise, scratches and the like. Disclosure of Invention In view of the foregoing, embodiments of the present application provide an image restoration method, apparatus, electronic device, and storage medium, so as to overcome or at least partially solve the foregoing problems. In a first aspect of an embodiment of the present application, an image restoration method is disclosed, the method including: Acquiring an image to be repaired; processing the image to be repaired according to an image repair model to obtain a repaired high-definition image; The image restoration model is trained through a plurality of training phases, and a loss function of one training phase in the plurality of training phases is a loss function aiming at optimizing structural information of an image. In a second aspect of an embodiment of the present application, an image restoration apparatus is disclosed, the apparatus including: The acquisition module is used for acquiring the image to be repaired; The restoration module is used for processing the image to be restored according to the image restoration model to obtain a restored high-definition image; The image restoration model is trained through a plurality of training phases, and a loss function of one training phase in the plurality of training phases is a loss function aiming at optimizing structural information of an image. In a third aspect of the embodiment of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the image restoration method according to the first aspect of the embodiment of the present application when the processor executes the computer program. In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the image restoration method according to the first aspect of the embodiments of the present application. The embodiment of the application has the following advantages: In the embodiment of the application, the image to be repaired is processed according to the image repair model to obtain the repaired high-definition image, and the image repair model is trained through a plurality of training stages, and the loss function of one training stage in the plurality of training stages is the loss function aiming at optimizing the structural information of the image, so that the image repair model can repair the structural information (such as block noise and other structural defects) in the image to be repaired. Therefore, according to the repaired high-definition image obtained by repairing the image repairing model, the repairing quality of the image can be remarkably improved in terms of structural defects, and high-quality image repairing is realized. Drawings In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. FIG. 1 is a flow chart of steps of an image restoration method according to an embodiment of the present application; FIG. 2 is a schematic structural diagram of an image restoration model according to an embodiment of the present application; FIG. 3 is a schematic structural diagram of another imag