EP-4371066-B1 - IMAGE PROCESSING METHOD, DEVICE, ELECTRONIC APPARATUS AND STORAGE MEDIUM

EP4371066B1EP 4371066 B1EP4371066 B1EP 4371066B1EP-4371066-B1

Inventors

ZHANG, JIANXING
LIU, Zikun
XIE, Zheng
YANG, JIAN
CHUN, Hyungju
WEN, WEI

Dates

Publication Date: 20260506
Application Date: 20221021

Claims (7)

A computer- implemented image processing method, comprising: acquiring an input image (S110); detecting a target area in the input image (S120); obtaining a feature map of the target area by extracting image features of the target area, rearranging the feature blocks in the feature map in a feature space, wherein the feature map is discretized in feature blocks of the same size and arranged along a channel dimension according to a preset arrangement rule; determining at least one of a level of importance of the rearranged feature blocks and a correlation between different feature blocks; and weighting and combining the rearranged feature blocks, based on the at least one of the level of importance, of the rearranged feature blocks and the correlation between different feature blocks; obtaining the output image after the target area is processed based on the weighted and combined feature blocks and the obtained feature map, wherein the determining of the correlation between the different feature blocks comprises: acquiring semantic layout information of the target area and texture direction field information of the feature blocks of the feature map, wherein the texture direction field information comprises texture direction field consistency information; and determining the correlation between the different feature blocks, based on the at least one of the semantic layout information and the texture direction field information, wherein the acquiring of the semantic layout information comprises: obtaining absolute semantic layout information by parsing the target area; obtaining relative semantic layout information by detecting key points of the target area; and obtaining the semantic layout information by encoding the obtained absolute semantic layout information and the relative semantic layout information, wherein the obtaining of the absolute semantic layout information comprises: obtaining a face parsing map by parsing the target area; and obtaining the absolute semantic layout information by performing a blur processing on the face parsing map, wherein the obtaining of the relative semantic layout information comprises: detecting the key points of the target area; selecting a first base point and a second base point from among the detected key points; and obtaining the relative semantic layout information by mapping a vector including at least one point in the target area and the first base point, to a reference vector including the first base point and the second base point, wherein the obtaining of the output image after the target area is processed based on the weighted and combined feature blocks and the feature map comprises: fusing the semantic layout information of the target area to the feature map; obtaining a reconstructed feature map by recovering the weighted and combined feature blocks to initial positions of the feature blocks thereof in the feature map; fusing the reconstructed feature map with the feature map fused with the semantic layout information; and obtaining the output image after the target area is processed based on the fused feature map.
The image processing method of claim 1, wherein the determining of the level of importance of the at least one rearranged feature block comprises: acquiring at least one of quality degradation level information and texture direction field information of at least one feature block of the feature map; and determining the level of importance of the at least one feature block, based on the at least one of the quality degradation level information and the texture direction field information, wherein quality degradation level information is information on a degree of image quality of the input image.
The image processing method of claim 2, wherein the texture direction field information comprises texture direction field strength information.
The image processing method of claim 2, wherein the acquiring of the quality degradation level information comprises: reducing the input image to a predetermined size; predicting quality degradation levels of different areas of the input image reduced to the predetermined size; and quantizing the predicted quality degradation levels to acquire the quality degradation level information
The image processing method of claim 1 or 2, wherein the acquiring of the texture direction field information comprises: acquiring a gradient field corresponding to the at least one feature block of the feature map; and obtaining the texture direction field information by applying expansion convolutions with different expansion rates to the gradient field.
An image processing device, comprising: at least one storage (2201) configured to store one or more computer executable instructions, and at least one processor (2202) configured to execute the one or more instructions stored in the storage to: acquire an input image; detect a target area in the input image; obtain a feature map of the target area by extracting image features of the target area, rearrange the feature blocks in the feature map in a feature space, wherein the feature map is discretized in feature blocks of the same size and arranged along a channel dimension according to a preset arrangement rule; determining at least one of a level of importance of the rearranged feature blocks and a correlation between different feature blocks; weighting and combining the rearranged feature blocks, based on the at least one of the level of importance, of the rearranged feature blocks and the correlation between different feature blocks; obtaining the output image after the target area is processed based on the weighted and combined feature blocks and the obtained feature map, wherein the determining of the correlation between the different feature blocks comprises: acquire semantic layout information of the target area and texture direction field information of the feature blocks of the feature map, wherein the texture direction field information comprises texture direction field consistency information; and determine the correlation between the different feature blocks, based on the at least one of the semantic layout information and the texture direction field information, wherein the acquiring of the semantic layout information comprises: obtain absolute semantic layout information by parsing the target area; obtain relative semantic layout information by detecting key points of the target area; and obtain the semantic layout information by encoding the obtained absolute semantic layout information and the relative semantic layout information, wherein the obtaining of the absolute semantic layout information comprises: obtain a face parsing map by parsing the target area; and obtain the absolute semantic layout information by performing a blur processing on the face parsing map, wherein the obtaining of the relative semantic layout information comprises: detect the key points of the target area; select a first base point and a second base point from among the detected key points; and obtain the relative semantic layout information by mapping a vector including at least one point in the target area and the first base point, to a reference vector including the first base point and the second base point, wherein the obtaining of the output image after the target area is processed based on the weighted and combined feature blocks and the feature map comprises: fuse the semantic layout information of the target area to the feature map; obtain a reconstructed feature map by recovering the weighted and combined feature blocks to initial positions of the feature blocks thereof in the feature map; fuse the reconstructed feature map with the feature map fused with the semantic layout information; and obtain the output image after the target area is processed based on the fused feature map.
A computer-readable storage medium configured to store instructions which when executed by at least one processor (2202), cause the at least one processor (2202) to execute the image processing method of any one of claims 1 to 5.

Description

[Technical Field] The disclosure relates to an artificial intelligent field, and more particularly to an image processing method and device, an electronic apparatus and a storage medium. [Background Art] The camera function of an intelligent terminal, for example a smart phone, is an important function. With the popularity and changing frequency of the intelligent terminal, the camera function of the intelligent terminal is more and more powerful, the image resolution is larger and larger, and the imaging details are clearer and clearer. The image is an objective reflection of the real world, and the imaging quality is a core index to evaluate the camera function. Therefore, improving the image imaging quality has become an important goal pursued by many manufacturers. However, due to the limited physical structure of the intelligent terminal, there is still a certain gap between the imaging quality of the intelligent terminal and the professional camera. Especially under a dark light condition, due to insufficient illumination, the image photographed by the intelligent terminal will also have serious quality degradation (e.g., texture loss), especially for the portrait part in the image, and the serious quality degradation will greatly affect the usage experience of a user. In some related art solutions, a certain preset filtering operator may be used to achieve the enhancement of the image texture. However, the recovering effect for texture details is poor if only the filtering is used to improve the image quality degradation, thus, the improvement of the image quality is very limited. In view of this, a better technology for improving or correcting the image quality degradation is needed. Conventional image processing methods and devices including using artificial intelligence models are disclosed in KIM TAEOH ET AL: "Block-Attentive Subpixel Prediction Networks for Computationally Efficient Image Restoration", IEEE ACCESS, IEEE, USA, vol. 9, 24 June 2021 (2021-06-24), pages 90881-90895; CN 112 990 053 A and 01 01 ET AL: "Underwater Image Co-Enhancement With Correlation Feature Matching and Joint Learning", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, IEEE, USA, vol. 32, no. 3, 20 April 2021 (2021-04-20), pages 1133-1147. [Disclosure] [Technical Solution] The invention is defined by the appended claims. The description that follows is subjected to this limitation. Any disclosure lying outside the scope of said claims is only intended for illustrative as well as comparative purposes. The technical solutions provided by the embodiments of the present disclosure at least bring the following advantageous effects: according to the image processing method and device of the embodiment of the present disclosure, by rearranging feature blocks in the feature map in a feature space, and obtaining an output image after the target area is processed based on the rearranged feature blocks and the feature map, so that: as for a large area of texture missing areas in the target area, details thereof may be effectively restored, thereby improving the image quality degradation better. It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and may not limit the present disclosure. [Description of Drawings] The drawings herein are incorporated into the specification and constitute a part of the specification, show example embodiments conforming to the present disclosure, and together with the specification to explain the principle of the present disclosure, and do not constitute an improper limitation of the present disclosure. The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which: FIG. 1 is a flowchart of an image processing method according to an embodiment;FIG. 2 is a schematic diagram showing spatial rearrangement of feature blocks, according to an embodiment;FIG. 3 is a schematic diagram showing enlarging a receptive field through rearrangement of the feature blocks, according to an embodiment;FIG. 4 is a schematic diagram showing operations of processing of a target area according to an embodiment;FIG. 5 is a schematic diagram showing a procedure and principle of the processing of the target area according to an embodiment;FIG. 6 is a schematic diagram of operations of an image processing method according to an embodiment;FIG. 7 is a schematic diagram of a texture direction field according to another embodiment;FIG. 8 is a schematic diagram showing texture direction field information is used for guiding texture generation;FIG. 9 is a schematic diagram of a texture trend extraction operation;FIGs. 10A-10B are schematic diagrams showing enhancement and propagation of a direction field;FIG. 11 is a schematic diagram showing a structure of a machine learning model a