Search

CN-115700726-B - Image processing method and device, training method and computer readable storage medium

CN115700726BCN 115700726 BCN115700726 BCN 115700726BCN-115700726-B

Abstract

Image processing method and device, training method and computer readable storage medium. The disclosure provides an image processing method, which comprises the steps of obtaining an input image and N-1 level momentum items, wherein N is a positive integer and N is more than 2, generating N level initial feature images with resolution ranging from high to low based on the input image, performing i level iterative back projection processing based on i+1th level initial feature images and i level momentum items to generate i level updated feature images, and generating an output image based on i=1, 2. The disclosure also provides an image processing device, a training method of the neural network, and a non-transitory computer readable storage medium.

Inventors

  • NA YANBO
  • LU YUNHUA

Assignees

  • 京东方科技集团股份有限公司

Dates

Publication Date
20260508
Application Date
20210730

Claims (14)

  1. 1. An image processing method, comprising: Acquiring an input image and N-1 levels of momentum items, wherein N is a positive integer and N is more than 2; Generating N levels of initial feature images with resolution ranging from high to low based on the input image; For the N-level initial feature images, performing an i-level iterative backprojection process based on the i+1-level initial feature images and the i-level motion term to generate an i-level updated feature image; Generating an output image based on the updated feature image of level 1; the iterative back projection processing of each level comprises a downsampling processing, a connecting processing, an upsampling processing, a first superposition processing and a second superposition processing; Downsampling of the ith layer includes downsampling based on an input of an iterative backprojection process of the ith layer to generate a downsampled output of the ith layer; The connection processing of the ith level comprises the steps of performing connection operation based on the downsampled output of the ith level and the initial characteristic image of the (i+1) th level, and generating joint output of the ith level; The i-th level upsampling process includes generating an i-th level upsampled output based on the i-th level joint output; the first superposition processing of the ith level includes superposing the first superposition input of the ith level and the up-sampling output of the ith level to generate a first superposition output of the ith level; Superposing the input of the iterative back projection processing of the ith level with the first superposition output of the ith level to generate the output of the iterative back projection processing of the ith level; The iterative back projection processing of the j+1th level is nested between the downsampling processing of the j level and the coupling processing of the j level, the input of the iterative back projection processing of the j+1th level comprises the downsampling output of the j level, wherein j=1, 2, and N-2, the iterative back projection processing of at least one level is continuously executed for a plurality of times, the input of the subsequent iterative back projection processing comprises the output of the previous iterative back projection processing, the first superposition input of the first superposition processing in the subsequent iterative back projection processing comprises the first superposition output of the first superposition processing in the previous iterative back projection processing, the first superposition input in the first iterative back projection processing comprises the dynamic item of the level, and the updated characteristic image of the 1 st level comprises the output of the last iterative back projection processing of the 1 st level.
  2. 2. The image processing method according to claim 1, wherein the generating of the i-th level joint output based on the i-th level downsampled output and the i+1-th level initial feature image specifically includes: Taking the downsampled output of the i < th > level as an input to the iterative backprojection processing of the i < th > level to generate an output of the iterative backprojection processing of the i < th > level; and linking the output of the i+1th-level iterative backprojection process with the i+1th-level initial feature image to generate the i-level joint output.
  3. 3. The image processing method according to claim 1, wherein generating N levels of initial feature images arranged from high to low in resolution based on the input image, comprises: and carrying out analysis processing of N different levels on the input image to respectively generate the N levels of initial characteristic images with the resolution ranging from high to low.
  4. 4. The image processing method according to claim 1, wherein generating an output image based on the updated feature image of level 1, comprises: And converting the 1 st-level updated feature image to generate the output image.
  5. 5. The image processing method according to claim 1, wherein generating N levels of initial feature images arranged from high to low in resolution based on the input image, comprises: Taking the input image as a middle input image of a 1 st level, and downsampling the input image to respectively generate middle input images of a2 nd level to an N th level, which are arranged from high resolution to low resolution; Analyzing and processing the intermediate input image of each level to generate an input characteristic image of each level, wherein the input characteristic image of the Nth level is used as an initial characteristic image of the Nth level; For each of the first N-1 levels, sequentially performing downsampling and analysis processing on the intermediate input image of the level to generate an intermediate feature image, connecting the intermediate feature image of the level with the initial feature image of the next level, upsampling the image generated after connection, superposing the upsampled image with the momentum item of the level to generate a first momentum item, and superposing the first momentum item with the input feature image of the level to generate the initial feature image of the level.
  6. 6. The image processing method according to claim 5, wherein the iterative backprojection processing of each hierarchy is performed M times in succession, M being an integer greater than 1, each iterative backprojection processing including a downsampling process, a concatenation process, an upsampling process, a first superimposition process, and a second superimposition process; The downsampling process in the mth iterative back projection process of the ith level comprises the steps of downsampling based on the input of the mth iterative back projection process of the ith level to generate downsampled output of the mth iterative back projection process of the ith level, wherein the initial feature image of the ith level comprises the input of the 1 st iterative back projection process of the ith level, and the input of each iterative back projection process after the 1 st iterative back projection process of the ith level comprises the output of the previous iterative back projection process; the coupling processing in the mth iterative back projection processing of the ith level comprises the step of carrying out coupling operation on the basis of the downsampled output of the mth iterative back projection processing of the ith level and the output of the mth iterative back projection processing of the (i+1) th level so as to generate an mth compensation characteristic image of the ith level; The up-sampling processing in the mth iterative back projection processing of the ith level comprises up-sampling based on the mth compensation characteristic image of the ith level to generate up-sampling output of the mth iterative back projection processing of the ith level; The first superposition processing in the 1 st iteration back projection processing of the i < th > level comprises performing superposition operation based on the up-sampling output of the 1 st iteration back projection processing of the i < th > level and the first motion item of the i < th > level to generate a first superposition output of the 1 st iteration back projection processing of the i < th > level; The second superposition processing in the 1 st iteration back projection processing of the ith level comprises the steps of carrying out superposition operation based on the first superposition output of the 1 st iteration back projection processing of the ith level and the initial characteristic image of the ith level to generate second superposition output of the 1 st iteration back projection processing of the ith level, wherein the second superposition processing in each iteration back projection processing of the 1 st level comprises the steps of carrying out superposition operation based on the first superposition output of the iteration back projection processing and the second superposition output of the last iteration back projection processing to generate second superposition output of the iteration back projection processing, and the second superposition output of each iteration back projection processing serves as output of the iteration back projection processing; Where m=1, 2, M, and outputting a second superposition of the last iteration back projection processing of the 1 st level as an updated characteristic image of the 1 st level.
  7. 7. The image processing method according to claim 5, wherein generating an output image based on the updated feature image of level 1, comprises: and converting the updated characteristic image of the 1 st level, and superposing the image generated after conversion with the input image to generate the output image.
  8. 8. The image processing method according to claim 1 or 5, wherein generating the N-level initial feature images of which resolution is arranged from high to low based on the input image, comprises: concatenating the input image with a random noise image to generate a joint input image; And carrying out analysis processing of N different levels on the combined input image so as to respectively generate the N levels of initial characteristic images with the resolution ranging from high to low.
  9. 9. The image processing method according to any one of claims 1 to 7, wherein, of the N-level initial feature images, a resolution of a1 st-level initial feature image is highest, and the resolution of the 1 st-level initial feature image is the same as that of the input image; The resolution of the initial feature image of the previous level is an integer multiple of the resolution of the initial feature image of the subsequent level.
  10. 10. A training method of a neural network is characterized in that the neural network comprises an analysis network, an iterative back projection processing network and an output network, and comprises the following steps: acquiring a training input image and preset N-1 momentum items, wherein N is a positive integer and N is more than 2; Processing the training input image by using the analysis network to generate training initial characteristic images of N layers with high-to-low arrangement resolution; performing iterative back projection processing of the ith level based on the training initial feature image of the (i+1) th level and the dynamic term of the (i) th level by using the iterative back projection processing network to generate a training updated feature image of the (i) =1, 2, & gt, N-1; Generating a training output image based on the training update feature image of level 1 using the output network; calculating a loss value of the neural network through a loss function based on the training output image, and correcting parameters of the neural network according to the loss value of the neural network; the iterative back projection processing of each level comprises a downsampling processing, a connecting processing, an upsampling processing, a first superposition processing and a second superposition processing; Downsampling of the ith layer includes downsampling based on an input of an iterative backprojection process of the ith layer to generate a downsampled output of the ith layer; The connection processing of the ith level comprises the steps of performing connection operation based on the downsampled output of the ith level and the initial characteristic image of the (i+1) th level, and generating joint output of the ith level; The i-th level upsampling process includes generating an i-th level upsampled output based on the i-th level joint output; the first superposition processing of the ith level includes superposing the first superposition input of the ith level and the up-sampling output of the ith level to generate a first superposition output of the ith level; Superposing the input of the iterative back projection processing of the ith level with the first superposition output of the ith level to generate the output of the iterative back projection processing of the ith level; The iterative back projection processing of the j+1th level is nested between the downsampling processing of the j level and the coupling processing of the j level, the input of the iterative back projection processing of the j+1th level comprises the downsampling output of the j level, wherein j=1, 2, and N-2, the iterative back projection processing of at least one level is continuously executed for a plurality of times, the input of the subsequent iterative back projection processing comprises the output of the previous iterative back projection processing, the first superposition input of the first superposition processing in the subsequent iterative back projection processing comprises the first superposition output of the first superposition processing in the previous iterative back projection processing, the first superposition input in the first iterative back projection processing comprises the dynamic item of the level, and the updated characteristic image of the 1 st level comprises the output of the last iterative back projection processing of the 1 st level.
  11. 11. The training method of claim 10, wherein the loss function comprises a mean square error between a training standard image corresponding to the training input image and the training output image.
  12. 12. An image processing apparatus, comprising: the image acquisition module is configured to acquire an input image and preset N-1 momentum items, wherein N is a positive integer; An image processing module configured to generate N levels of initial feature images arranged from high to low in resolution, N being a positive integer and N >2, based on the input image, perform an i-th level iterative backprojection process based on the i+1-th level initial feature image and the i-th level motion term for the N levels of initial feature images to generate an i-th level updated feature image, i=1, 2..; the iterative back projection processing of each level comprises a downsampling processing, a connecting processing, an upsampling processing, a first superposition processing and a second superposition processing; Downsampling of the ith layer includes downsampling based on an input of an iterative backprojection process of the ith layer to generate a downsampled output of the ith layer; The connection processing of the ith level comprises the steps of performing connection operation based on the downsampled output of the ith level and the initial characteristic image of the (i+1) th level, and generating joint output of the ith level; The i-th level upsampling process includes generating an i-th level upsampled output based on the i-th level joint output; the first superposition processing of the ith level includes superposing the first superposition input of the ith level and the up-sampling output of the ith level to generate a first superposition output of the ith level; Superposing the input of the iterative back projection processing of the ith level with the first superposition output of the ith level to generate the output of the iterative back projection processing of the ith level; The iterative back projection processing of the j+1th level is nested between the downsampling processing of the j level and the coupling processing of the j level, the input of the iterative back projection processing of the j+1th level comprises the downsampling output of the j level, wherein j=1, 2, and N-2, the iterative back projection processing of at least one level is continuously executed for a plurality of times, the input of the subsequent iterative back projection processing comprises the output of the previous iterative back projection processing, the first superposition input of the first superposition processing in the subsequent iterative back projection processing comprises the first superposition output of the first superposition processing in the previous iterative back projection processing, the first superposition input in the first iterative back projection processing comprises the dynamic item of the level, and the updated characteristic image of the 1 st level comprises the output of the last iterative back projection processing of the 1 st level.
  13. 13. An image processing apparatus, comprising: a memory and a processor, the memory having stored thereon a computer program, wherein the computer program when executed by the processor implements the image processing method of any of claims 1 to 9 or the training method of any of claims 10 to 11.
  14. 14. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the image processing method of any of claims 1 to 9 or the training method of any of claims 10 to 11.

Description

Image processing method and device, training method and computer readable storage medium Technical Field The disclosure relates to the technical field of display, in particular to an image processing method and device, a neural network training method and a non-transitory computer readable storage medium. Background Currently, artificial neural network-based deep learning techniques have made tremendous progress in fields such as image classification, image capturing and searching, face recognition, age, and speech recognition. The advantage of deep learning is that very different technical problems can be solved with a relatively similar system with a generic architecture. Convolutional neural networks (Convolutional Neural Network, CNN) are artificial neural networks that have been developed and have attracted widespread attention in recent years, CNN is a special way of image recognition, and belongs to very efficient networks with forward feedback. Disclosure of Invention The present disclosure proposes an image processing method, an image processing apparatus, a training method of a neural network, a non-transitory computer-readable storage medium. In a first aspect, the present disclosure provides an image processing method, including: Acquiring an input image and N-1 levels of momentum items, wherein N is a positive integer and N is more than 2; Generating N levels of initial feature images with resolution ranging from high to low based on the input image; For the N-level initial feature images, performing an i-level iterative backprojection process based on the i+1-level initial feature images and the i-level motion term to generate an i-level updated feature image; an output image is generated based on the updated feature image of level 1. In some embodiments, the iterative backprojection process for each level includes a downsampling process, a concatenation process, an upsampling process, a first superimposition process, and a second superimposition process; Downsampling of the ith layer includes downsampling based on an input of an iterative backprojection process of the ith layer to generate a downsampled output of the ith layer; The connection processing of the ith level comprises the steps of performing connection operation based on the downsampled output of the ith level and the initial characteristic image of the (i+1) th level, and generating joint output of the ith level; The i-th level upsampling process includes generating an i-th level upsampled output based on the i-th level joint output; the first superposition processing of the ith level includes superposing the first superposition input of the ith level and the up-sampling output of the ith level to generate a first superposition output of the ith level; Superposing the input of the iterative back projection processing of the ith level with the first superposition output of the ith level to generate the output of the iterative back projection processing of the ith level; The iterative back projection processing of the j+1th level is nested between the downsampling processing of the j level and the coupling processing of the j level, the input of the iterative back projection processing of the j+1th level comprises the downsampling output of the j level, wherein j=1, 2, and N-2, the iterative back projection processing of at least one level is continuously executed for a plurality of times, the input of the subsequent iterative back projection processing comprises the output of the previous iterative back projection processing, the first superposition input of the first superposition processing in the subsequent iterative back projection processing comprises the first superposition output of the first superposition processing in the previous iterative back projection processing, the first superposition input in the first iterative back projection processing comprises the dynamic item of the level, and the updated characteristic image of the 1 st level comprises the output of the last iterative back projection processing of the 1 st level. In some embodiments, the method further comprises generating a joint output of the ith hierarchy based on the downsampled output of the ith hierarchy and the initial feature image of the (i+1) th hierarchy, specifically comprising: Taking the downsampled output of the i < th > level as an input to the iterative backprojection processing of the i < th > level to generate an output of the iterative backprojection processing of the i < th > level; and linking the output of the i+1th-level iterative backprojection process with the i+1th-level initial feature image to generate the i-level joint output. In some embodiments, generating N levels of initial feature images arranged from high to low resolution based on the input image comprises: and carrying out analysis processing of N different levels on the input image to respectively generate the N levels of initial characteristic images with the resoluti