KR-102963982-B1 - Apparatus and method for fast adaptation of image inpainting neural network

KR102963982B1KR 102963982 B1KR102963982 B1KR 102963982B1KR-102963982-B1

Abstract

A method for high-speed adaptation of an image inpainting neural network according to one embodiment comprises: a step of inputting a single input image in which a portion of the area is masked into an image inpainting neural network to generate a first output image in which the masked portion of the area is restored; a step of setting the first output image as a temporary target image; a step of randomly masking a portion of the first output image and inputting it into the image inpainting neural network to generate a second output image in which the masked portion of the area is restored, calculating a loss value by comparing the second output image with the temporary target image, and repeating the process of updating the parameters of the image inpainting neural network based on the calculated loss value a predetermined number of times; and a step of calculating a loss value by comparing the output image output from the image inpainting neural network with the original image of the single input image after the predetermined number of repetitions is completed, and further updating the parameters of the image inpainting neural network based on the loss value.

Inventors

최재원
이강태
최형기
김태현
이은혜

Assignees

주식회사 케이티

Dates

Publication Date: 20260511
Application Date: 20210315

Claims (11)

In a device for high-speed adaptation of an image inpainting neural network, An image inpainting neural network that receives a single input image with some regions masked, restores the masked regions of the single input image, and generates an output image; A masking processing unit that repeats the process of randomly masking a portion of an output image generated by the image inpainting neural network and inputting it into the image inpainting neural network a predetermined number of times; A loss calculation unit that sets an output image restored from one input image through the image inpainting neural network as a temporary target image, and calculates a loss value by comparing the temporary target image with an output image generated by restoring from an image input to the image inpainting neural network after masking processing in the masking processing unit; and It includes an optimization unit that updates the parameters of the image inpainting neural network based on the above loss value, and The above loss calculation unit calculates a loss value by comparing the output image output from the image inpainting neural network after the completion of the above predetermined number of iterations with the original image of the one input image, and The above optimization unit is a device that further updates the parameters of the image inpainting neural network based on the loss value obtained by comparing the original image.
In paragraph 1, The above masking processing unit is, An apparatus characterized by generating a random binary mask composed of values of 0 or 1, and multiplying the output image by the random binary mask to randomly mask a portion of the output image.
In paragraph 1, The image inpainting neural network, the masking processing unit, the loss calculation unit, and the optimization unit are, An apparatus characterized by repeating the same processing using a single input image with some areas masked and its original image, using multiple different input images with some areas masked and their original images.
In paragraph 1, A device characterized by fine-tuning the image inpainting neural network using only another input image in which some regions are masked during test time.
In paragraph 4, The above fine-tuning is, The above image inpainting neural network receives another input image in which some regions are masked, restores the masked regions of the input image to generate an output image, and The masking processing unit repeats the process of randomly masking a portion of an output image output from the image inpainting neural network and inputting it into the image inpainting neural network a predetermined number of times. The loss calculation unit sets the output image restored from the other input image as the target image, and calculates a loss value by comparing the target image with the output image generated by restoring the image input to the image inpainting neural network after masking processing in the masking processing unit. A device characterized in that the optimization unit updates the parameters of the image inpainting neural network based on the loss value calculated by the loss calculation unit.
In a method for fast adaptation of an image inpainting neural network, A step of inputting a single input image in which some regions are masked into an image inpainting neural network to generate a first output image in which the masked regions are restored; A step of setting the first output image as a temporary target image; A step of repeating the process of randomly masking a portion of the first output image and inputting it into the image inpainting neural network to generate a second output image in which the masked portion is restored, calculating a loss value by comparing the second output image with the temporary target image, and updating the parameters of the image inpainting neural network based on the calculated loss value a predetermined number of times; and A method comprising the step of calculating a loss value by comparing an output image output from the image inpainting neural network with an original image of one input image after the above-mentioned number of iterations is completed, and further updating the parameters of the image inpainting neural network based on the loss value.
In paragraph 6, The step of repeatedly performing the above is, A method characterized by generating a random binary mask composed of values of 0 or 1, and multiplying the first output image by the random binary mask to randomly mask a portion of the first output image.
In paragraph 6, A method characterized by further including the step of generating the first output image, the step of setting the temporary target image, the step of repeating the steps of performing the repetition and the step of additionally updating, using a plurality of different input images in which some regions are masked and their original images.
In paragraph 6, A method characterized by further including the step of fine-tuning the image inpainting neural network using only another input image in which some regions are masked during the test time.
In Paragraph 9, The above fine-tuning step is, A step of inputting another input image in which some regions are masked into an image inpainting neural network to generate a third output image in which some masked regions are restored; A step of setting the above third output image as the first target image; and A method comprising the step of repeating the process of randomly masking a portion of a portion of a third output image and inputting it into an image inpainting neural network to generate a fourth output image in which the masked portion is restored, calculating a loss value by comparing the fourth output image with the first target image, and updating the parameters of the image inpainting neural network based on the calculated loss value a predetermined number of times.
A computer program recorded on a recording medium as a computer program that executes a method according to any one of paragraphs 6 through 10 through a computer system.

Description

Apparatus and method for fast adaptation of image inpainting neural network The present invention relates to image inpainting neural network technology, and more specifically, to an apparatus and method for high-speed adaptation of an image inpainting neural network. Image inpainting refers to an image post-processing technique that restores damaged or empty areas within an image by visually filling them in naturally. It is commonly used to remove unnecessary areas, such as subtitles or advertisements, and to composite images so that the removed areas appear natural compared to the surrounding areas. Recently, along with the advancement of deep learning technology, the performance of image inpainting has been improving dramatically, and it is now widely used not only for removing small areas like subtitles but also for removing relatively large objects within images, such as people, cars, and backgrounds. In general, a large amount of training data is required to train artificial intelligence neural networks. Training data consists of pairs of data and their corresponding correct labels; thus, a significant amount of data is necessary to train these networks. Training an image inpainting neural network also requires a large amount of training data. In other words, a large number of pairs of images and their corresponding correct labels are needed. Securing such training data is not easy, and even if a large amount of data is obtained, training the neural network takes a considerable amount of time. Meanwhile, after training an artificial intelligence neural network, fine-tuning is performed during the test phase to finely adjust the network's parameters. In other words, the parameters of the neural network are updated by further training the pre-trained network. The parameters of an image inpainting neural network can also be updated through fine-tuning. Just as with training, a large amount of training data is required for fine-tuning, but securing such data is difficult. Furthermore, fine-tuning a pre-trained image inpainting neural network requires hundreds or even thousands of fine-tuning iterations. Consequently, there is a problem of it being time-consuming. FIG. 1 is a diagram showing the configuration of a high-speed adaptation device for high-speed adaptation of an image inpainting neural network according to one embodiment of the present invention. FIG. 2 is a diagram comparing the conventional learning and fine-tuning of an image inpainting neural network and the fast adaptation and fine-tuning of an image inpainting neural network according to an embodiment of the present invention in terms of neural network parameters. FIG. 3 is a flowchart illustrating a learning method for high-speed adaptation of an image inpainting neural network according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a method for fine-tuning a learned image inpainting neural network according to an embodiment of the present invention. The aforementioned objectives, features, and advantages will become clearer through the following detailed description in conjunction with the attached drawings, and accordingly, a person skilled in the art to which the present invention pertains will be able to easily implement the technical concept of the present invention. Furthermore, in describing the present invention, if it is determined that a detailed description of known technology related to the present invention may unnecessarily obscure the essence of the present invention, such detailed description will be omitted. Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings. FIG. 1 is a diagram showing the configuration of a device for high-speed adaptation of an image inpainting neural network according to one embodiment of the present invention (hereinafter referred to as a high-speed adaptation device). Referring to FIG. 1, the high-speed adaptation device according to the present embodiment includes an image inpainting neural network (120), a masking processing unit (130), a loss calculation unit (140), and an optimization unit (150). These can be implemented as a program, stored in memory, and executed by at least one processor. The fast adaptation device according to the present embodiment aims to move the parameters of the image inpainting neural network (120) to a location in the parameter space where fine-tuning can be performed well during test time, rather than a location where a large amount of training data can be processed well. The fast adaptation device according to the present embodiment applies a modified meta-learning algorithm to image inpainting. As a meta-learning technique, the present embodiment is based on Finn's MAML (Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, 2017), but is not limited thereto, and various meta-learning techniques (e.g., F0-MAML, MAML++, FO-MAML+L2F, Meg