US-12626329-B2 - Electronic device for upscaling image and method for controlling same

US12626329B2US 12626329 B2US12626329 B2US 12626329B2US-12626329-B2

Abstract

An example electronic device includes a memory; a display; and at least one processor operatively coupled to the memory and the display. The at least one processor may be configured to divide an input image into a plurality of divided images, acquire image characteristics included in each of the plurality of divided images, identify at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in a memory on the basis of the image characteristics, acquire a plurality of upscaled segmented images corresponding respectively to the plurality of divided images through the at least one deep learning model, merge the plurality of upscaled divided images to obtain an upscaled image, and display the upscaled image on the display.

Inventors

Bongsoo Jung
Bonghyuck KO

Assignees

SAMSUNG ELECTRONICS CO., LTD.

Dates

Publication Date: 20260512
Application Date: 20230508
Priority Date: 20210113

Claims (18)

1 . An electronic device comprising: memory storing instructions; a display; and at least one processor, comprising processing circuitry, operatively connected with the memory and the display, wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to: divide an input image into a plurality of divided images, wherein a first divided image the plurality of divided images includes a padding area which overlaps with an adjacent divided image, and a seize of the padding area is based on a complexity of the first divided image; obtain an image characteristic included in each of the plurality of divided images; identify, based on the image characteristic, at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in the memory; obtain, through the at least one deep learning model, a plurality of upscaled divided images respectively corresponding to the plurality of divided images; obtain an upscaled image by merging the plurality of upscaled divided images based on a border processing operation for upscaled images of the first divided image and the adjacent divided images using the padding area; and display the upscaled image on the display.
2 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to identify at least one of a number of the plurality of divided images or a size of the plurality of divided images based on at least one of an available size of the memory or a size of the input image.
3 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to obtain, as the image characteristic, at least one of image complexity of each of the plurality of divided images or whether a face image is included.
4 . The electronic device of claim 3 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to: identify, as a first deep learning model, a deep learning model to process a divided image not including a face image and having an image complexity equal to or larger than a set value among the plurality of dividing images; identify, as a second deep learning model, a deep learning model to process a divided image not including a face image and having an image complexity less than the set value among the plurality of dividing images; and identify, as a third deep learning model, a deep learning model to process a divided image including a face image among the plurality of dividing images.
5 . The electronic device of claim 1 , comprising a plurality of processors, each of the plurality of processors comprising processing circuitry and being configured to perform image upscaling through at least one of the plurality of deep learning models.
6 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to obtain an average of a pixel value of a padding area corresponding to the area of each of the plurality of upscaled divided images and a pixel value of a padding area corresponding to the area of a neighboring upscaled divided image of each of the plurality of upscaled divided images, as a pixel value of the area.
7 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to store, as metadata of the upscaled image, at least one of information about the image characteristic, information about the identified at least one deep learning model, or position information about the plurality of divided images.
8 . The electronic device of claim 1 , wherein the size of the padding area increases as a complexity of the first divided image increases.
9 . A method for controlling an electronic device, the method comprising: dividing an input image into a plurality of divided images, wherein a first divided image of the plurality of divided images includes a padding area which overlaps with an adjacent divided image, and a size of the padding area is based on a complexity of the first divided image; obtaining an image characteristic included in each of the plurality of divided images; identifying, based on the image characteristic, at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in a memory; obtaining, through the at least one deep learning model, a plurality of upscaled divided images respectively corresponding to the plurality of divided images; obtaining an upscaled image by merging the plurality of upscaled divided images based on a border processing operation for upscaled images of the first divided image and the adjacent divided images using the padding area; and displaying the upscaled image.
10 . The method of claim 9 , wherein the dividing includes identifying at least one of a number of the plurality of divided images or a size of the plurality of divided images based on at least one of an available size of the memory or a size of the input image.
11 . The method of claim 9 , wherein obtaining the image characteristic obtains, as the image characteristic, at least one of image complexity of each of the plurality of divided images or whether a face image is included.
12 . The method of claim 11 , wherein the identifying identifies, as a first deep learning model, a deep learning model to process a divided image not including a face image and having an image complexity equal to or larger than a set value among the plurality of divided images; identifies, as a second deep learning model, a deep learning model to process a dividing image not including a face image and having an image complexity less than the set value among the plurality of dividing images; and identifies, as a third deep learning model, a deep learning model to process a divided image including a face image among the plurality of divided images.
13 . The method of claim 9 , wherein obtaining the plurality of upscaled divided images is performed by each of a plurality of processors using at least one of the plurality of deep learning models when the electronic device includes the plurality of processors.
14 . The method of claim 9 , wherein obtaining the plurality of upscaled divided images comprising obtaining, as a pixel value of the area, an average of a pixel value of a padding area corresponding to the area of each of the plurality of upscaled divided images and a pixel value of a padding area corresponding to the area of a neighboring upscaled divided image of each of the plurality of upscaled divided images.
15 . The method of claim 9 , further comprising storing, as metadata of the upscaled image, at least one of information about the image characteristic, information about the identified at least one deep learning model, or position information about the plurality of divided images.
16 . The method of claim 9 , wherein the size of the padding area increases as a complexity of the first divided image increases.
17 . An electronic device comprising: memory storing instructions; a display; and at least one processor, comprising processing circuitry, operatively connected with the memory and the display, wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to: obtain an image characteristic for each area of an input image, divide the input image into a plurality of divided images based on the image characteristic for each area, wherein a first divided image of the plurality of divided images includes a padding area which overlaps with an adjacent divided image, and a size of the padding area is based on a complexity of the first divided image, identify at least one deep learning model to process of each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in the memory based on the image characteristic for each area, obtain a plurality of upscaled divided images respectively corresponding to the plurality of divided images through the at least one deep learning model, obtain an upscaled image by merging the plurality of upscaled divided images based on a border processing operation for upscaled images of the first divided image and the adjacent divided images using the padding area, and display the upscaled image on the display.
18 . The electronic device of claim 17 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to: obtain at least one of an image complexity of the input image or whether a face image is included as the image characteristic, identify at least one of a size, shape, or number of each of the plurality of divided images based on the image complexity or whether the face image is included, and divide the input image into the plurality of divided images based on at least one of the size, shape, or number of each of the plurality of divided images.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a continuation of PCT/KR2021/020319, designating the United States, filed Dec. 30, 2021, in the Korean Intellectual Property Receiving Office, which claims priority to Korean Patent Application No. 10-2021-0004768, filed on Jan. 13, 2021, in the Korean Intellectual Property Office. The disclosures of each of these applications are incorporated herein in their entireties. BACKGROUND Field The disclosure relates to an electronic device upscaling an image and a method for controlling the same. Description of Related Art Conventional image upscaling includes rule-based interpolation algorithms, such as nearest neighbor, bilinear, bicubic, and lanczos. These techniques operate in a manner to interpolate intermediate pixel values necessary for upscaling by referring to information about neighboring pixels based on their respective predetermined rules. With the development of technology, image upscaling has recently been performed using a deep learning model for image upscaling. The deep learning model for image upscaling operates in a manner to select the optimal deep learning model parameters by learning a large number of input images and applying selected parameters to the input image. These deep learning models may be divided into on-device modeling with a deep learning model engine in a mobile device and server-based modeling techniques with a deep learning model engine in a server. SUMMARY Conventional image upscaling schemes by rule-based interpolation algorithms, such as nearest neighbor, bilinear, bicubic, and lanczos, simply resize images but do not enhance the quality, causing image quality degradation of upscaled images. A scheme of obtaining a high-resolution upscaled image using a deep learning model used to address such issues requires significant computational load and memory and thus use of such a deep learning model in a mobile device may be limited. For example, when the deep learning model is used in a mobile device, if the resolution of the input image is large, the deep learning model requires a large amount of memory and may thus significantly slow down or, in a worst scenario case, shut down, operation of the mobile device, thereby restricting the size of the input image. Accordingly, upscaling using a deep learning model is processed mostly on a server, and the processing results are received by a terminal. Example embodiments of the disclosure provide an electronic device and a method for controlling the same, which may efficiently perform image upscaling using a deep learning model on a mobile device based on the characteristics of the input image and resources of the mobile device. According to various example embodiments, an electronic device may include a memory, a display, and at least one processor operatively connected with the memory and the display. The at least one processor may be configured to divide an input image into a plurality of divided images, obtain an image characteristic included in each of the plurality of divided images, identify at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in the memory, based on the image characteristic, obtain a plurality of upscaled divided images respectively corresponding to the plurality of divided images through the at least one deep learning model, obtain an upscaled image by merging the plurality of upscaled divided images, and display the upscaled image on the display. According to various example embodiments, a method for controlling an electronic device may include dividing an input image into a plurality of divided images, obtaining an image characteristic included in each of the plurality of divided images, identifying at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in a memory, based on the image characteristic, obtaining a plurality of upscaled divided images respectively corresponding to the plurality of divided images through the at least one deep learning model, obtaining an upscaled image by merging the plurality of upscaled divided images, and displaying the upscaled image. According to various example embodiments, an electronic device may include a memory, a display, and at least one processor operatively connected with the memory and the display. The at least one processor may be configured to obtain an image characteristic for each area of an input image, divide the input image into a plurality of divided images based on the image characteristic for each area, identify at least one deep learning model to process of each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in the memory based on the image characteristic for each area, obtain a plurality of upscaled divided images respectively corresponding to