US-12621424-B2 - Electronic apparatus and control method thereof

US12621424B2US 12621424 B2US12621424 B2US 12621424B2US-12621424-B2

Abstract

An electronic apparatus is disclosed. The electronic apparatus includes: a memory storing at least one instruction and at least one processor, comprising processing circuitry, individually and/or collectively, configured to: identify a foreground region and a background region included in an input image based on a depth map corresponding to the input image, and generate a novel view image by converting a viewpoint based on the foreground region; identify side effect prediction information including at least one from among whether an object of less than or equal to a specified thickness is included in the input image or an object density degree based on the depth map, and generate the novel view image by controlling a viewpoint movement path based on the identified side effect prediction information.

Inventors

Seungho Park
Changsu Kim
Keunsoo KO
Daesung Cho
Youngsu Moon
Jinyoung JUN
Seunggyun WOO
Younghoon JEONG

Assignees

SAMSUNG ELECTRONICS CO., LTD.
KOREA UNIVERSITY RESEARCH AND BUSINESS FOUNDATION

Dates

Publication Date: 20260505
Application Date: 20241216
Priority Date: 20230912

Claims (20)

1 . An electronic apparatus, comprising: a memory storing at least one instruction; and at least one processor, comprising processing circuitry, individually and/or collectively, configured to: identify a foreground region and a background region included in an input image based on a depth map corresponding to the input image, identify side effect prediction information based on at least one from among whether an object of less than or equal to a specified thickness is included in the input image or whether an object including an object density degree greater than or equal to a threshold value is included in the input image, and generate a novel view image by converting a viewpoint based on the foreground region and by controlling a viewpoint movement path based on the identified side effect prediction information, wherein the viewpoint movement path is reduced based on the object of less than or equal to the specified thickness being identified as being included in the input image or the object including the object density degree greater than or equal to the threshold value being identified as being included in the input image.
2 . The electronic apparatus of claim 1 , wherein the viewpoint movement path is reduced to less than a threshold range based on the object of less than or equal to the specified thickness being identified as being included in the input image or the object including the object density degree greater than or equal to the threshold value being identified as being included in the input image.
3 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: control, based on the object of less than or equal to the specified thickness being identified as being included in the input image based on the depth map, reduction of the viewpoint movement path in proportion to an area ratio which the object of less than or equal to the specified thickness occupies in the input image.
4 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: obtain a depth map applied with an opening operation by applying the opening operation to the depth map, and identify a region including the object of less than or equal to the specified thickness based on difference information between the depth map and the depth map applied with the opening operation.
5 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: calculate a standard deviation of depth values of pixels excluding a depth boundary region within a specified window by applying the specified window to the depth map, and perform a depth map refinement based on the calculated standard deviation being greater than or equal to a threshold value.
6 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: generate, based on the object density degree being identified as greater than or equal to the threshold value based on the depth map, the novel view image by controlling reduction of the viewpoint movement path to less than a threshold range.
7 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: adjust an occlusion region and a context region based on a boundary complexity indicating the object density degree, wherein the occlusion region includes a region not exposed by the foreground region from a current viewpoint and exposed at movement of the viewpoint, and wherein the context region is a region adjacent to the occlusion region.
8 . The electronic apparatus of claim 7 , wherein at least one processor, individually and/or collectively, is configured to: identify the boundary complexity based on a number of neighboring boundaries included within the context region, and reduce a width of at least one from among the occlusion region and the context region based on the number of neighboring boundaries being greater than or equal to a threshold number.
9 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: refine the depth map by substituting a depth value of a region corresponding to the object of less than or equal to the specified thickness with a depth value of a surrounding pixel having the most similar color information with a target pixel.
10 . The electronic apparatus of claim 1 , wherein at least one processor, individually and/or collectively, is configured to: refine the depth map by substituting a depth value of other than the region corresponding to the object of less than or equal to the specified thickness with a median value of depth values of pixels within a specified window.
11 . A method of controlling an electronic apparatus, the method comprising: identifying a foreground region and a background region included in an input image based on a depth map corresponding to the input image; and generating a novel view image by converting a viewpoint based on the foreground region, wherein the generating a novel view image comprises: identifying side effect prediction information based on at least one from among whether an object of less than or equal to a specified thickness is included in the input image or whether an object including an object density degree greater than or equal to a threshold value is included in the input image; and generating the novel view image by controlling a viewpoint movement path based on the identified side effect prediction information, wherein the viewpoint movement path is reduced based on the object of less than or equal to the specified thickness being identified as being included in the input image or the object including the object density degree greater than or equal to the threshold value being identified as being included in the input image.
12 . The method of claim 11 , wherein the viewpoint movement path is reduced to less than a threshold range based on the object of less than or equal to the specified thickness being identified as being included in the input image or the object including the object density degree greater than or equal to the threshold value being identified as being included in the input image.
13 . The method of claim 11 , wherein the generating the novel view image comprises: controlling, based on the object of less than or equal to the specified thickness being identified as being included in the input image based on the depth map, reduction of the viewpoint movement path in proportion to an area ratio which the object of less than or equal to the specified thickness occupies in the input image.
14 . The method of claim 11 , wherein the generating the novel view image comprises: obtaining a depth map applied with an opening operation by applying the opening operation to the depth map; and identifying a region included with the object of less than or equal to the specified thickness based on difference information between the depth map and the depth map applied with the opening operation.
15 . The method of claim 11 , further comprises: calculating a standard deviation of depth values of pixels excluding a depth boundary region within a specified window by applying the specified window to the depth map, and performing a depth map refinement based on the calculated standard deviation being greater than or equal to a threshold value.
16 . A non-transitory computer-readable storage medium storing computer commands which when executed individually and/or collectively by at least one processor of an electronic apparatus, cause the electronic apparatus to perform operations comprising: identifying a foreground region and a background region included in an input image based on a depth map corresponding to the input image; and generating a novel view image by converting a viewpoint based on the foreground region, wherein the generating a novel view image comprises: identifying side effect prediction information based on at least one from among whether an object of less than or equal to a specified thickness is included in the input image or whether an object including an object density degree greater than or equal to a threshold value is included in the input image; and generating the novel view image by controlling a viewpoint movement path based on the identified side effect prediction information, wherein the viewpoint movement path is reduced based on the object of less than or equal to the specified thickness being identified as being included in the input image or the object including the object density degree greater than or equal to the threshold value being identified as being included in the input image.
17 . The non-transitory computer-readable storage medium of claim 16 , wherein the viewpoint movement path is reduced to less than a threshold range based on the object of less than or equal to the specified thickness being identified as being included in the input image or the object including the object density degree greater than or equal to the threshold value being identified as being included in the input image.
18 . The non-transitory computer-readable storage medium of claim 16 , wherein the generating the novel view image comprises: controlling, based on the object of less than or equal to the specified thickness being identified as being included in the input image based on the depth map, reduction of the viewpoint movement path in proportion to an area ratio which the object of less than or equal to the specified thickness occupies in the input image.
19 . The non-transitory computer-readable storage medium of claim 16 , wherein the generating the novel view image comprises obtaining a depth map applied with an opening operation by applying the opening operation to the depth map; and identifying a region included with the object of less than or equal to the specified thickness based on difference information between the depth map and the depth map applied with the opening operation.
20 . The non-transitory computer-readable storage medium of claim 16 , wherein the operations further comprises: calculating a standard deviation of depth values of pixels excluding a depth boundary region within a specified window by applying the specified window to the depth map, and performing a depth map refinement based on the calculated standard deviation being greater than or equal to a threshold value.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of International Application No. PCT/KR2024/010288 designating the United States, filed on Jul. 17, 2024, in the Korean Intellectual Property Receiving Office and claiming priority to Korean Patent Application No. 10-2023-0121410, filed on Sep. 12, 2023, in the Korean Intellectual Property Office, the disclosures of each of which are incorporated by reference herein in their entireties. BACKGROUND Field The disclosure relates to an electronic apparatus and a control method thereof, and for example, to an electronic apparatus that generates novel view images and the control method thereof. Description of Related Art Electronic devices of various types are being developed and supplied with developments in electronic technology. Specifically, display apparatuses used in various locations such as homes, offices, and public spaces have continuously been under development for several recent years. A novel view synthesis technique is a technique which generates an image of a viewpoint different from a viewpoint of a 2-dimensional (2D) image which is obtained through a monocular camera. Using this technique, a consecutive video of a novel view image according to movement of a viewpoint trajectory for a single image may be generated. SUMMARY According to an example embodiment, an electronic apparatus includes: a memory storing at least one instruction and at least one processor, comprising processing circuitry, individually and/or collectively, configured to: identify a foreground region and a background region included in an input image based on a depth map corresponding to the input image, and generate a novel view image by converting a viewpoint based on the foreground region; identify side effect prediction information including at least one from among whether an object of less than or equal to a specified thickness is included in the input image or an object density degree based on the depth map, and generate the novel view image by controlling a viewpoint movement path based on the identified side effect prediction information. According to an example embodiment, at least one processor, individually and/or collectively, may be configured to: generate, based on the object of less than or equal to the specified thickness being identified as included in the input image or the object density degree being identified as greater than or equal to a threshold value based on the depth map, the novel view image by controlling reduction of the viewpoint movement path to less than a threshold range. According to an example embodiment, at least one processor, individually and/or collectively, may be configured to: control, based on the object of less than or equal to the specified thickness being identified as included in the input image based on the depth map, reduction of the viewpoint movement path in proportion to an area ratio which the object of less than or equal to the specified thickness occupies in the input image. According to an example embodiment, at least one processor may be configured to: obtain a depth map applied with an opening operation by applying the opening operation to the depth map, and identify a region included with the object of less than or equal to the specified thickness based on difference information between the depth map and the depth map applied with the opening operation. According to an example embodiment, at least one processor may be configured to: calculate a standard deviation of depth values of pixels excluding a depth boundary region within the window by applying the window to the depth map, and perform a depth map refinement based on the calculated standard deviation being greater than or equal to a threshold value. According to an example embodiment, at least one processor may be configured to: generate, based on the object density degree being identified as greater than or equal to a threshold value based on the depth map, the novel view image by controlling reduction of the viewpoint movement path to less than a threshold range. According to an example embodiment, at least one processor may be configured to: adjust an occlusion region and a context region based on a boundary complexity indicating the object density degree, and the occlusion region may be a region not exposed by the foreground region from a current viewpoint and exposed at movement of the viewpoint, and the context region may be a region adjacent with the occlusion region. According to an example embodiment, at least one processor may be configured to: identify the boundary complexity based on a number of neighboring boundaries included within the context region, and reduce a width of at least one from among the occlusion region and the context region based on the number of neighboring boundaries being greater than or equal to a threshold number. According to an example embodiment, at least one processor may be configured to: refine the d