KR-102962920-B1 - METHOD AND APPARATUS FOR COLORING TARGET IMAGES BASED ON AT LEAST ONE COLORED REFERENCE IMAGE

Abstract

An image coloring method is provided, which acquires a pre-colored reference image serving as a standard for coloring and a plurality of target images to be colored, and colors the target images based on the reference image. The coloring of the target images is performed under coloring conditions that include maintaining the line art of each target image and coloring each target image only with the colors used to color the reference image.

Inventors

• 김나영
• 이명기
• 이준수

Assignees

• 네이버웹툰 유한회사

Dates

Publication Date

20260511

Application Date

20240417

Claims (20)

1. In an image coloring method performed by a computer system, A step of acquiring a pre-colored reference image serving as a standard for coloring and a plurality of target images to be colored; and A step of coloring the target images based on the reference image under coloring conditions comprising maintaining the line art of each target image of the target images and coloring each target image only with the colors used for coloring the reference image Includes, The above target images are a series of images that constitute the animation, and The step of coloring the above target images is, A step of obtaining first segments, which are regions separated in the line drawing of the reference image above, and second segments, which are regions separated in the line drawing of each target image above; A step of performing segment matching between the first segments and the second segments; and A step of coloring each target image based on the reference image based on the result of the above segment matching. Includes, The step of acquiring the second segment above is, For a first target image comprising at least two colors among the above target images and at least one colored region, a step of separating the first target image into color layers based on said colors; A step of obtaining line area layers by distinguishing line areas expressed in a predetermined reference color and other colors for each of the above color layers, and obtaining area area layers by distinguishing area areas including the colored areas; A step of merging the above line area layers to create a line area merge layer; and A step of dividing the first target image into segments based on the line region merging layer, and obtaining the second segments by merging at least some of the segments using the surface region layers. An image coloring method including
2. In paragraph 1, The step of coloring the above target images is, An image coloring method that, as the above reference image and each target image are input, colors the target images using a pre-trained transformation model to extract a color for coloring each target image from the above reference image.
3. In paragraph 1, An image coloring method in which each of the above target images uses two or more colors and includes at least one colored area.
4. In paragraph 3, The colored area of each of the above target images represents a corresponding area between the target images, and An image coloring method based on the above reference image, wherein the colored area of each target image in the above target images is colored with the same at least one color.
5. delete
6. delete
7. In paragraph 1, The step of creating the above line area merge layer is, A step of converting the line area expressed in the other color included in the line area merging layer into the reference color. An image coloring method including
8. delete
9. In paragraph 1, The above reference image includes a colored image and a line art image, and The step of acquiring the first segments above is, A step of obtaining line area layers and area area layers by distinguishing line areas and area areas for the above-mentioned line drawing image; and A step of obtaining the first segments of the reference image based on a line region merge layer formed by merging the line region layers. An image coloring method including
10. In Article 1 or Article 9, The distinction between the above-mentioned line region and the above-mentioned surface region is, It is performed using a pre-trained detector model to separate lines and surfaces from an image, and An image coloring method in which, if the saturation of a surface region separated using the detector model exceeds a predetermined threshold, the surface region separated using the detector model is separated into the line region.
11. In paragraph 1, The step of performing the above segment matching is, A step of extracting local features and semantic features from each of the above target images; A step of generating a feature set for each target image by fusing the local features and the semantic features; A step of grouping the set of features based on the second segments; and A step of generating a second segment representative vector for each of the second segments based on the grouped set of features. An image coloring method including
12. In Paragraph 11, The step of generating the second segment representative vector above is, An image coloring method that generates a second segment representative vector for each second segment based on the average of features grouped as corresponding to each second segment among the features included in the above feature set.
13. In Paragraph 11, The step of performing the above segment matching is, An image coloring method that performs segment matching by determining a first segment among the first segments that matches each of the second segments based on the correlation between a second segment representative vector for each of the second segments and a first segment representative vector representing each of the first segments' characteristics.
14. In paragraph 1, The above reference images are plural and include a first reference image and a second reference image, and The above coloring step is, An image coloring method based on the similarity between the first reference image and the segments within each target image and the similarity between the second reference image and the segments within each target image, wherein at least some of the segments of the target images are colored based on the first reference image and the remainder of the segments of the target images are colored based on the second reference image.
15. In Paragraph 14, The above coloring step is, A step of calculating a similarity probability between a second target image among the target images or a third segment within the second target image and each of the plurality of reference images; A step of determining a reference image among the above reference images that has the maximum similarity probability; and Step of extracting a color for coloring a third segment within the second target image from the above-determined reference image An image coloring method including
16. In paragraph 15, Each of the above reference images and each of the above target images includes their respective frame numbers, and The step of calculating the above similarity probability is, An image coloring method that calculates the similarity probability as the difference between the frame number of each reference image and the frame number of the second target image is larger.
17. In paragraph 1, Each of the above target images includes at least one line area with a color different from at least one reference color and at least one colored area, and A step of dividing the line regions or colored regions of different colors of each target image into at least one fourth segment; A step of determining the center of the fourth segment; A step of determining a segment of the reference image corresponding to the fourth segment based on the distance between the center of each of the segments of the reference image and the center of the fourth segment; and Step of coloring the regions of the target images corresponding to the fourth segment with the color colored in the segment of the determined reference image. An image coloring method that further includes
18. In paragraph 1, A step of identifying a polychromatic region based on whether at least one of the following exists: when two or more colors are used in line segments separated by a predetermined unit within the reference image, when one of the colors used to color the reference image is colored only in a predetermined number or fewer of pixels, and when there exists an area colored with a specific pixel value within the reference image; and A step of notifying the user who colored the reference image of the identified polychromatic area An image coloring method that further includes
19. A program recorded on a computer-readable recording medium for executing the method of claim 1 on the computer system.
20. In a computer system that performs an image coloring method, At least one processor implemented to execute readable instructions in the above computer system Includes, The above-mentioned at least one processor is, Acquire a pre-colored reference image that serves as the standard for coloring, and multiple target images that are the subject of coloring, and Under coloring conditions including maintaining the line art of each of the above target images and coloring each of the above target images only with the colors used for coloring the above reference image, the target images are colored based on the above reference image, and The above target images are a series of images that constitute the animation, and The above at least one processor, in coloring the target images, Acquiring first segments, which are regions separated in the line drawing of the reference image above, and second segments, which are regions separated in the line drawing of each target image above, Segment matching is performed between the first segments and the second segments, and Based on the result of the above segment matching, each target image is colored based on the above reference image, and The above at least one processor, in acquiring the second segment, For a first target image comprising at least two colors among the above target images and at least one colored area, the first target image is separated into color layers based on the colors, and For each of the above color layers, line regions expressed in a predetermined reference color and other colors are distinguished to obtain line region layers, and surface regions including the colored regions are distinguished to obtain surface region layers. The above line area layers are merged to create a line area merge layer, and A computer system that divides the first target image into segments based on the line region merging layer, and obtains the second segments by merging at least some of the segments using the surface region layers.

Description

Method and apparatus for coloring target images based on at least one colored reference image The present disclosure relates to a method and apparatus for coloring target images based on at least one pre-colored reference image, and relates to a method and apparatus for automatically coloring target images, which are a series of images constituting an animation, based on a pre-colored reference image under predetermined coloring conditions. Animation consists of a series of consecutive images. Each of these images can constitute a frame of the animation. To produce such an animation, it is required to color each of the images that make up the animation. In other words, animation can be produced by coloring the line drawings that constitute each frame. The coloring of line art images can be performed, for example, using reference images that have been pre-colored by the artist. The artist can produce the animation by directly coloring the line art images one by one while looking at these pre-colored reference images. Coloring these line art images is time-consuming as it must be performed repeatedly on a large number of images, and it requires a significant level of skill from the worker. Furthermore, the fatigue caused by this repetitive work and the high difficulty of the coloring process often lead to errors in the resulting colors. This results in prolonged animation production and high costs. Korean Published Patent No. 10-2009-0111939 (published on October 28, 2009) discloses a method and apparatus for separating an image into foreground and background, a method and apparatus for separating an image into foreground and background and replacing the background, and a computer-readable recording medium storing a program for executing the method. The information described above is for illustrative purposes only and may include content that does not constitute part of the prior art and may not include what the prior art would present to a person skilled in the art. FIG. 1 illustrates an image coloring method for coloring a plurality of target images based on a pre-colored reference image according to one embodiment. FIG. 2 shows a computer system that performs an image coloring method according to one embodiment. FIG. 3 is a flowchart illustrating an image coloring method for coloring a plurality of target images based on a pre-colored reference image according to one embodiment. FIG. 4 is a flowchart illustrating a method for coloring a target image based on a reference image, based on segment matching between a reference image and a target image, according to one example. FIG. 5 is a flowchart illustrating a method of segmenting an image into multiple segments according to one example. Figure 6 is a flowchart illustrating a method for performing segment matching between a reference image and a target image according to one example. FIG. 7 is a flowchart illustrating an image coloring method for coloring multiple target images based on each reference image in the case where there are multiple reference images, according to one example. FIG. 8 is a flowchart illustrating a method for performing post-processing on colored target images according to one example. Figure 9 is a flowchart illustrating a method for verifying a reference image according to one example. Figure 10 shows a transformation model used to color target images based on a reference image. Figure 11 shows a target image to be colored, according to one example. Figure 12 illustrates a method of segmenting an image into multiple segments according to one example. FIG. 13 shows an image segmented into multiple segments according to one example. FIG. 14 shows a detector module that detects line regions and plane regions from a color layer of an image according to one example. FIG. 15 illustrates a method for generating a feature set for an input image by extracting local features and semantic features from an input image according to one example. FIG. 16 illustrates a method for generating a segment tokenization set by grouping a feature set for an input image based on segment information, according to one example. FIG. 17 illustrates a method for performing segment matching using a segment vector corresponding to each segment according to one example. FIG. 18 illustrates a method of coloring input images using a plurality of reference images according to one example. FIG. 19 illustrates a method for transferring the color of a reference image to a colored target image by performing post-processing on the colored target images according to one example. FIG. 20 illustrates a method for identifying colored parts in a reference image according to one example. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. FIG. 1 illustrates an image coloring method for coloring a plurality of target images based on a pre-colored reference image according to one embodiment. Referring to FIG. 1,