US-12620192-B2 - Method and apparatus for improving object image

US12620192B2US 12620192 B2US12620192 B2US 12620192B2US-12620192-B2

Abstract

Provided are a method and an apparatus for restoring an object image, capable of restoring an image naturally by detecting positions of landmarks of an object in a bounding-box detected from an input image, performing warping to align the object at a central position or a reference position on the basis of the landmarks, improving the image using a learning model learned from the aligned object image, performing inverse warping for rotating the improved object image in an original direction or at an original angle, and inserting the inversely-warped object image into the input image. In addition, provided are a method and an apparatus for restoring an object image, capable of detecting positions of landmarks of an object in a bounding-box detected from an input image, performing pose estimation for a side object on the basis of the landmarks, and improving an image using a learning model learned from a side object image corresponding to the pose estimation result.

Inventors

Jaeseob SHIN
Sungul RYOO
Sehoon SON
Hyeongduck KIM
Hyosong KIM
Kyunghwan KO

Assignees

PIXTREE, INC.

Dates

Publication Date: 20260505
Application Date: 20221205
Priority Date: 20200605

Claims (11)

1 . An apparatus for improving an object image comprising: one or more communication buses; and a plurality of software modules configured to perform communication using the one or more communication buses and process at least one operation for improving an object image, the plurality of software modules comprising: an input module that receives an input image; a bounding-box detecting module that detects a plurality of bounding-boxes from the input image; an object recognizing module that recognizes an object in each of the bounding-boxes; a background extracting module that extracts a background image from the input image; an object image improving module that detects landmarks that are main features of each object, performs warping for aligning an object position at a central position or a reference position on the basis of the landmarks to generate a warped object image, performs inference so as to improve the warped object image using a first pre-learned object learning model specialized for improving at least one of normalized object images to generate an improved object image, and performs inverse warping for inverting the improved object image to the object position of the input image, wherein the inverse warping includes reversing of an angular alignment performed during the warping by rotating the improved object image in an original direction or at an original angle to generate an inversely-warped object image; a background image improving module that performs inference so as to improve the background image using a second pre-learned background image learning model specialized for improving the background image to generate an improved background image, wherein the second pre-learned learning model is different from the first pre-learned learning model; a segmentation module that segments the inversely-warped object image; a blending module that blends the segmented inversely-warped object image and the improved background image to generate a blended image; and an output module that outputs the blended image as an output image.
2 . An apparatus for improving an object image comprising: one or more communication buses; and a plurality of software modules configured to perform communication using the one or more communication buses and process at least one operation for improving an object image, the plurality of software modules comprising: an input module that receives a plurality of objects displayed in a bounding-box; a landmark detecting module that detects landmarks that are main features of each of the objects; a warping module that performs warping for aligning an object position at a central position or a reference position on the basis of the landmarks to generate a warped object image; a resizing module that resizes the warped object image to a preset target size to generate a resized warped object image; an inference module that receives the resized warped object image and performs inference so as to improve the resized warped object image in a resized bounding box using a pre-learned object learning model to generate an improved object image; an inversely-resizing module that receives the improved object image from the inference module and inversely resizes the improved object image obtained by improving the resized warped object image by the inference module to an original size to generate an inversely-resized improved object image; an inverse warping module that performs inverse warping for inverting the inversely-resized improved object image to the object position of the bounding-box to generate an inversely-warped object image; and an output module that applies the inversely-warped object image to the bounding-box.
3 . The apparatus according to claim 2 , wherein the warping module aligns a reference feature point included in the landmarks, of the object in the bounding-box, to be positioned on a predetermined fixed line.
4 . The apparatus according to claim 3 , wherein in aligning the reference feature point to be positioned on the predetermined fixed line, in a case where it is determined that the object image is a front object image that faces the front, the warping module performs the warping by rotating the front object image clockwise or counterclockwise only in a roll direction among 6 axes.
5 . The apparatus according to claim 3 , wherein in aligning the reference feature point to be positioned on the predetermined fixed line, the warping module performs the warping by rotating the object image clockwise or counterclockwise only in a roll direction among 6 axes.
6 . The apparatus according to claim 2 , wherein the warping module finds main feature points of the landmarks, extracts a midpoint of an upper horizontal axis (x′) that connects upper features among the main feature points, extracts a midpoint of a lower horizontal axis that connects lower features among the main feature points, connects the midpoint of the upper horizontal line (x′) and the midpoint of the lower horizontal axis line with a vertical axis line (y′), and performs the warping on the basis of the upper horizontal axis line (x′) and the vertical axis line (y′) that connects the midpoint of the upper horizontal axis line (x′) and the midpoint of the lower horizontal axis line.
7 . The apparatus according to claim 6 , wherein the warping module performs length correction corresponding to an aspect ratio of the object for each of the upper horizontal axis line (x′) and the vertical axis line (y′) that connects the midpoint of the upper horizontal axis line (x′) and the midpoint of the lower horizontal axis line, compares the upper horizontal axis line (x′) with the vertical axis line (y′) that connects the midpoint of the upper horizontal axis line (x′) and the midpoint of the lower horizontal axis line in which the length correction is reflected, determines a larger axis as a reliable axis as a result of the comparison, and performs the warping by performing rotation on the basis of the reliable axis.
8 . The apparatus according to claim 2 , wherein the inference module improves, in a case where the warped object image is a front object image that faces the front, the quality of the warped object image using a restoring model learned on the basis of the front object image.
9 . The apparatus according to claim 8 , further comprising: a pose estimating module that determines, in a case where it is determined that the warped object image needs to be rotated in a yaw direction or a pitch direction among the 6 axes in order to face the front, that the object image is a side object image that faces a side, and performs pose estimation for the object of the side object image to estimate an object angle; and a parameter selecting module that selects a parameter corresponding to the object angle.
10 . The apparatus according to claim 2 , wherein the inference module improves, in a case where the warped object image is a side object image that faces a side, the quality of the warped object image using a restoring model learned on the basis of the side object image.
11 . A computer-implemented method for improving an object image, comprising: receiving a plurality of objects displayed in a bounding-box; detecting landmarks that are main feature of each of the objects; performing warping for aligning an object position at a central position or a reference position on the basis of the landmarks to generate a warped object image; resizing the warped object image to a preset target size to generate the resized warped object image; receiving the resized warped object image and performing inference so as to improve the resized warped object image using a pre-learned object learning model in a resized bounding box to generate an improved object image; receiving the improved object image from be performing of the inference and inversely resizing the improved object image generated by improving the resized warped object image to an original size to generate an inversely-resized improved object image; performing inverse warping for inverting the inversely-resized improved object image to the object position of the bounding-box to generate an inversely-warped object image; and applying the inversely-warped object image to the bounding-box.

Description

TECHNICAL FIELD The present disclosure relates to a method and an apparatus for restoring an object image. BACKGROUND ART The contents described herein merely provide background information related to the present inventive concept, and do not constitute the prior art. In general, techniques for restoring a low-resolution image to a high-resolution image are classified according to the number of input images used for restoration or restoring techniques. The techniques are classified into a single image super-resolution restoring technique and a continuous image super-resolution restoring technique, according to the number of input images. Generally, the single image super-resolution image restoring technique has a faster processing speed than that of the continuous image super-resolution image restoring technique, but has a low quality of image restoration since information necessary for restoration is insufficient. Since the continuous image super-resolution image restoring technique uses various features extracted from a plurality of consecutively acquired images, the quality of the restored image is superior to that of the single image super-resolution image restoring technique, but its algorithm is complicated and the amount of computation is large, thereby making it difficult to perform real-time processing. As the restoring techniques, a technique using interpolation, a technique using edge information, a technique using frequency characteristics, or a technique using machine learning such as deep learning is used. The technique using learning such as deep learning is used. The technique using interpolation has a high processing speed, but has a disadvantage of blurring of edge parts. The technique using edge information has a high processing speed and maintains the sharpness of edge parts, but has a disadvantage of a visually noticeable restoration error in a case where an edge direction is incorrectly estimated. The technique using frequency characteristics maintains the sharpness of edge parts using high-frequency components similar to the technique using edge information, but has a disadvantage of occurrence of a ringing artifact near a boundary. Finally, the technique using machine learning such as example-based explanation or deep learning has the highest quality of restored images, but its processing speed is very slow. As described above, among the various high-resolution image restoring techniques, the continuous image super-resolution image restoring technique may be applied to fields necessary for a digital zoom function using the existing interpolation method, and can provide images of superior quality compared to the interpolation-based image restoring technique. However, the existing super-resolution image restoring technique is limited in its application to electro-optical equipment that requires limited resources and real-time processing due to a large amount of computation. The existing single image-based super-resolution image restoring technique capable of real-time processing has a problem in that performance is significantly reduced compared to the continuous image-based restoring technique in a case where an image needs to be enlarged with a high magnification of 2 times or more. DISCLOSURE Technical Problem An object of the present inventive concept is to provide a method and an apparatus for restoring an object image capable of detecting a bounding-box including an object that is defined in advance, detecting landmarks of the object in the detected bounding-box, performing warping for aligning the object to be positioned at a central position or a reference position on the basis of the landmarks, improving the image using a learning model learned on the basis of the aligned object image, performing inverse warping for rotating the improved image in an original direction or at an original angle, and inserting the result into an input image to restore a natural image. In addition, another object of the present inventive concept is to provide a method and an apparatus for restoring an object image capable of performing pose estimation on an object in a bounding-box detected from an input image, and improving the image using a learning model learned from a side object image corresponding to a result of the pose estimation. Technical Solution According to an aspect of the present inventive concept, there is provided an apparatus for improving an object image including: an input section that receives an input image; a bounding-box detecting section that detects a plurality of bounding-boxes from the input image; an object recognizing section that recognizes an object in each of the bounding-boxes; a background extracting section that extracts a background image from the input image; an object image improving section that detects landmarks that are main features of each object, performs warping for aligning an object position at a central position or a reference position on the basis o