CN-115937531-B - Image processing network, image resolution expansion method, device, equipment and medium

CN115937531BCN 115937531 BCN115937531 BCN 115937531BCN-115937531-B

Abstract

The invention discloses an image processing network, an image resolution expansion method, a display device, equipment and a medium. The image processing network of the embodiment comprises an image input end, a first convolution module, a characteristic extraction module, a first characteristic summation module and a sampling module, wherein the image input end is used for acquiring an original image to be processed, the first convolution module is connected with the image input end in series and used for carrying out convolution processing on the original image with a preset first convolution size to output a first image, the characteristic extraction module is connected with the first convolution module in series and used for carrying out residual error and pooled attention extraction on the first image to output a second image, the first characteristic summation module is used for carrying out characteristic summation processing on the second image input by the first input end and the first image input by the second input end, the sampling module is connected with the first characteristic summation module in series and used for carrying out sampling processing on the image output by the first characteristic summation module to generate a super-division image, and the image output end is used for outputting the super-division image generated by the sampling module.

Inventors

ZHU DAN
GAO YAN
DUAN RAN

Assignees

京东方科技集团股份有限公司

Dates

Publication Date: 20260508
Application Date: 20230103

Claims (9)

1. An image processing network based on a deep learning network, the image processing network comprising: the image input end is used for acquiring an original image to be processed; The first convolution module is connected with the image input end in series and is used for carrying out convolution processing on the original image with a preset first convolution size so as to output a first image; a feature extraction module, in series with the first convolution module, for residual and pooled attention extraction of the first image to output a second image, The first characteristic summation module comprises a first input end connected with the output end of the characteristic extraction module and a second input end connected with the first convolution module, and is used for carrying out characteristic summation processing on a second image input by the first input end and a first image input by the second input end; The sampling module is connected in series with the first characteristic summation module and is used for sampling the image output by the first characteristic summation module to generate a super-resolution image, The image output end is used for outputting the super-resolution image generated by the sampling module; The feature extraction module includes: A plurality of feature extraction units connected in series, wherein the feature extraction units positioned at the end part of one side of the series are connected with the first convolution module, and the feature extraction units output pooled attention images after feature extraction, and The second convolution unit is connected with the feature extraction unit at the end part of the other side of the series connection, and the other end of the second convolution unit is connected with the second input end of the first feature summation module and is used for convolving the pooled attention image input by the connected feature extraction unit to output a second image; wherein the feature extraction unit includes: a residual extraction unit for extracting residual from the image input to the residual extraction unit, and The pooling attention unit is connected with the residual extraction unit in series and is used for pooling and extracting features of the images generated by the residual extraction unit so as to output pooled attention images; the residual extraction unit includes: a first residual unit including a third input terminal as the residual extraction unit for performing extraction and normalization processing on an input image with a convolution check of 1*1; a second residual unit connected in series with the first residual unit for extracting and normalizing the input image with a 3*3 convolution check; The first residual summing unit is connected with the second residual unit in series and comprises a fourth input end connected with the second residual unit and a fifth input end connected with the first residual unit; a third residual unit connected to the first residual summing unit for extracting and normalizing the input image with 3*3 convolution checks, and And the second residual summing unit is connected with the third residual unit and comprises a sixth input end connected with the third residual unit and a seventh input end connected with the third input end, and is used for summing the image input by the sixth input end and the image input by the seventh input end, so that the generated image is output to the pooled attention unit.
2. The image processing network of claim 1, wherein the pooled attention unit comprises: The first pooling unit is used for pooling the images input by the residual extraction unit connected with the pooling attention unit so as to output first pooled images; The system comprises a plurality of parallel block self-attention units, a first pooling unit and a second pooling unit, wherein each block self-attention unit is connected with the output end of the first pooling unit and is used for respectively generating self-attention images with different preset window lengths; A second pooling unit connected to the output end of each of the block self-attention units for feature-combining the self-attention images with different preset window lengths to output a second pooled image, and And one end of the first characteristic product unit is connected with the second pooling unit, and the other end of the first characteristic product unit is connected with the output end of the second residual summing unit and is used for carrying out product according to the second pooled image and the first pooled image so as to generate a pooled attention image.
3. The image processing network of claim 2, wherein the block self-attention unit comprises: A blocking unit for blocking the first pooled image outputted by the first pooling unit with a preset window length to generate a blocked image containing a plurality of sub-images, A first shape reconstruction unit connected to the blocking unit for performing a size reset in a first matrix to generate a first shape reconstructed image, A second shape reconstruction unit connected to the blocking unit and connected in parallel to the first shape reconstruction unit for performing a size reset in a second matrix to generate a second shape reconstruction image, And the self-attention feature merging unit is connected with the first shape reconstruction unit and the output end of the second shape reconstruction unit and is used for carrying out feature merging according to the first shape reconstruction image and the second shape reconstruction image so as to generate a self-attention image corresponding to the preset window length.
4. An image processing network according to claim 3, wherein the self-attention feature combining unit comprises: the second characteristic product unit is respectively connected with the first shape reconstruction unit and the second shape reconstruction unit and is used for carrying out matrix product on the first shape reconstruction image and the second shape reconstruction image; the third shape reconstruction unit is connected with the second characteristic product unit and is used for reconstructing the shape of the image output by the second characteristic product unit through a fourth sub-reconstruction layer; And the merging unit is connected with the third shape reconstruction unit and is used for carrying out feature merging on the images output by the third shape reconstruction unit so as to generate a self-attention image corresponding to the preset window length.
5. The image processing network of claim 1, wherein the sampling module is any one of an up-sampling module, a bilinear interpolation sampling module, a bicubic interpolation sampling module, and a transposed convolution sampling module based on a pixel arrangement.
6. A method of image resolution extension, the method comprising: Detecting the resolution of an original image to be processed; setting a preset window length according to the resolution; inputting the set preset window length and the original image into a trained image processing network, wherein the image processing network is any one of claims 1-5, and And outputting the super-resolution image generated after being processed by the image processing network.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of claim 6 when executing the program.
8. A computer readable storage medium, on which a computer program is stored, which program, when being executed by a processor, implements the method according to claim 6.
9. A display device comprising a controller, wherein the controller implements the method of claim 6 when executing a program.

Description

Image processing network, image resolution expansion method, device, equipment and medium Technical Field The invention relates to the technical field of display. And more particularly, to an image processing network, an image resolution extension method, apparatus, device, and medium. Background As the display device is upgraded and iterated, the image resolution and video resolution perceived by the user is increasingly higher, e.g., from 480p to 720p, 2k, 4k. However, the existing image super-resolution model is mostly a large model with complex structure, the magnitude of which is at least hundreds of megameters and is difficult to be deployed at a terminal because of higher calculation amount required by high resolution, and if the image super-resolution model with smaller magnitude is adopted, the required super-resolution force is multiplied along with the increase of the resolution of an input video source because the image super-resolution model is fixed and can be deployed at the terminal. Therefore, integrating the image super-division model in the terminal equipment is particularly important to reasonably configure the image processing calculation force of the terminal. Disclosure of Invention The invention aims to provide an image processing network, an image resolution expansion method, an image resolution expansion device and an image resolution expansion medium, so as to solve at least one of the problems existing in the prior art. In order to achieve the above purpose, the invention adopts the following technical scheme: A first aspect of the present invention provides an image processing network based on a deep learning network, characterized in that the image processing network comprises: the image input end is used for acquiring an original image to be processed; The first convolution module is connected with the image input end in series and is used for carrying out convolution processing on the original image with a preset first convolution size so as to output a first image; a feature extraction module, in series with the first convolution module, for residual and pooled attention extraction of the first image to output a second image, The first characteristic summation module comprises a first input end connected with the output end of the characteristic summation module and a second input end connected with the first convolution module, and is used for carrying out characteristic summation processing on a second image input by the first input end and a first image input by the second input end; The sampling module is connected in series with the first characteristic summation module and is used for sampling the image output by the first characteristic summation module to generate a super-resolution image, And the image output end is used for outputting the super-resolution image generated by the sampling module. Further, the feature extraction module includes: A plurality of feature extraction units connected in series, wherein the feature extraction units positioned at the end part of one side of the series are connected with the first convolution module, and the feature extraction units output pooled attention images after feature extraction, and The second convolution unit is connected with the feature extraction unit at the end part of the other side of the series connection, and the other end of the second convolution unit is connected with the second input end of the first feature summation module and is used for convolving the pooled attention image input by the connected feature extraction unit to output a second image; wherein the feature extraction unit includes: a residual extraction unit for extracting residual from the image input to the residual extraction unit, and And the pooling attention unit is connected with the residual extraction unit in series and is used for pooling and extracting the characteristics of the images generated by the residual extraction unit so as to output pooled attention images. Further, the residual extraction unit includes: a first residual unit including a third input terminal as the residual extraction unit for performing extraction and normalization processing on an input image with a convolution check of 1*1; a second residual unit connected in series with the first residual unit for extracting and normalizing the input image with a 3*3 convolution check; The first residual summing unit is connected with the second residual unit in series and comprises a fourth input end connected with the second residual unit and a fifth input end connected with the first residual unit; a third residual unit connected to the first residual summing unit for extracting and normalizing the input image with 3*3 convolution checks, and And the second residual summing unit is connected with the third residual unit and comprises a sixth input end connected with the third residual unit and a seventh input end connected with the third input end, and is used for