EP-4738072-A1 - METHOD FOR USING REAL-WORLD OBJECT AS INPUT TOOL, AND ELECTRONIC DEVICE FOR CARRYING OUT SAME

Abstract

Provided is a method of using a real object as an input tool and an electronic device for performing the method. The electronic device comprises a camera for obtaining an image by photographing a real object and a user's hand in a real space, memory storing at least one instruction, and at least one processor configured to execute at least one instruction stored in the memory, wherein the at least one processor is configured to recognize a real object interacting with a user's hand from an obtained image, identify a pre-registered function which uses the recognized real object as an input tool, and perform the identified pre-registered function.

Inventors

• KO, Jaewoo
• RYU, Daeho
• KIM, JEONGWON
• SOHN, JUNIL
• YI, Sanghyun
• CHO, SUNGWOO

Assignees

• Samsung Electronics Co., Ltd.

Dates

Publication Date

20260506

Application Date

20240708

Claims (15)

1. An electronic device (200) comprising: a camera (210) configured to obtain an image by photographing a real object and a user's hand in a real space; memory (260) storing at least one instruction; and at least one processor (250) configured to execute the at least one instruction stored in the memory, wherein the at least one processor (250) is configured to: recognize the real object interacting with the user's hand from the obtained image, identify a pre-registered function which uses the recognized real object as an input tool, and perform the identified pre-registered function.
2. The electronic device (200) of claim 1, wherein the at least one processor (250) is configured to: detect the real object from the obtained image, divide the detected real object into a plurality of portions, detect at least one portion occluded by the user's hand from among the plurality of portions, and identify the pre-registered function based on the at least one occluded portion.
3. The electronic device (200) of any one of claims 1 and 2, wherein the at least one processor (250) is configured to: detect the real object from the obtained image, divide the detected real object into a plurality of portions, determine a position in which the user's hand is gripping the real object based on a result of the dividing, and identify the pre-registered function based on the determined position.
4. The electronic device (200) of any one of claims 2 and 3, wherein the at least one processor (250) is configured to: determine a gripping direction indicating toward which one of the plurality of portions the user's hand is gripping the real object, and identify the pre-registered function based on the determined gripping direction.
5. The electronic device (200) of any one of claims 1 to 4, wherein the at least one processor (250) is configured to: detect the real object from the obtained image, divide the detected real object into a plurality of portions, recognize a motion of the user's hand touching at least one of the plurality of portions, and identify the pre-registered function based on the recognized motion.
6. The electronic device (200) of any one of claims 1 to 5, wherein the at least one processor (250) is configured to: track a motion of the real object based on the obtained image, and perform a function for generating or deleting a virtual object based on the tracked motion.
7. The electronic device (200) of claim 6, wherein the at least one processor (250) is configured to: recognize at least one of a color and a size of at least a portion of the real object from the obtained image, determine at least one of a color and a size of the virtual object based on at least one of the color and the size of the at least a portion of the real object, and generate the virtual object having at least one of the determined color and size.
8. The electronic device (200) of any one of claims 6 and 7, wherein the at least one processor (250) is configured to: recognize a target object indicated by the real object from the obtained image, determine at least one of a color, a shape and a form of the virtual object based on at least one of a color, a shape and a form of the target object, and generate the virtual object having at least one of the determined color, shape and form.
9. The electronic device (200) of any one of claims 6 to 8, wherein the at least one processor (250) is configured to: classify the virtual object by using an artificial intelligence model having a virtual image corresponding to the virtual object as an input, determine a color mapped to a result of the classifying as a color of the virtual object, and generate the virtual object having the determined color.
10. The electronic device (200) of any one of claims 1 to 9, wherein the pre-registered function is registered by recognizing the real object from the image, registering the recognized real object as the input tool, and mapping an interaction between the registered real object and the user's hand to a predefined function.
11. A method of using a real object as an input tool of an electronic device, the method comprising: obtaining an image by photographing a real object and a user's hand in a real space (S310); recognizing the real object interacting with the user's hand from the obtained image (S320); identifying a pre-registered function which uses the recognized real object as an input tool (S330); and performing the identified pre-registered function (S340).
12. The method of claim 11, wherein: the recognizing of the real object interacting with the user's hand from the obtained image (S320) comprises: detecting the real object from the obtained image; dividing the detected real object into a plurality of portions; and detecting at least one portion occluded by the user's hand from among the plurality of portions, and the identifying of the pre-registered function which uses the recognized real object as the input tool (S330) comprises identifying the pre-registered function based on the at least one occluded portion.
13. The method of any one of claims 11 and 12, wherein: the recognizing of the real object interacting with the user's hand from the obtained image (S320) comprises: detecting the real object from the obtained image; dividing the detected real object into a plurality of portions; and determining a position in which the user's hand is gripping the real object based on a result of the dividing, and the identifying of the pre-registered function which uses the recognized real object as the input tool (S330) comprises identifying the pre-registered function based on the determined position.
14. The method of any one of claims 12 and 13, wherein: the recognizing of the real object interacting with the user's hand from the obtained image (S320) comprises determining a gripping direction indicating toward which one of the plurality of portions the user's hand is gripping the real object, and the identifying of the pre-registered function which uses the recognized real object as the input tool (S330) comprises identifying the pre-registered function based on the determined gripping direction.
15. A computer-readable recording medium having a program recorded thereon to cause a computer to perform the method of any one of claims 11 to 14.

Description

Technical Field The present disclosure relates to an electronic device for using a real-world object as an input tool and method thereof. More particularly, the present disclosure relates to an electronic device for detecting a real-world object that is not in communication with the electronic device and performing a predetermined function according to a result of the detecting and method thereof. Background Art Augmented reality (AR) is a technology for showing a virtual image by overlaying it on a physical environment space of a real world or a real-world object, and virtual reality (VR) is a technology for interacting with a non-existent virtual object or showing the virtual object independently in a virtual environment. Recently, AR devices (e.g., smart glasses) or VR devices that use the AR and VR technologies are being usefully used in daily life for e.g., information search, directions, camera shooting, games, etc. As touch operation is not possible, by nature, on the AR devices or VR devices, signal transmission and reception through the user's hand gesture or through a paired electronic device as an input means is used as an input interface to provide a service. Disclosure of Invention Solution to Problem In an embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a camera for obtaining an image by photographing a real-world object (hereinafter, also referred to as a real object) and a user's hand in a real space. The electronic device may include memory storing at least one instruction. The electronic device may include at least one processor configured to execute the at least one instruction stored in the memory. The at least one processor may recognize the real object interacting with the user's hand from the obtained image. The at least one processor may identify a pre-registered function which uses the recognized real object as an input tool. The at least one processor may perform the identified pre-registered function. In an embodiment of the present disclosure, a method of using a real object as an input tool of an electronic device may be provided. The method may include obtaining an image by photographing a real object and a user's hand in a real space. The method may include recognizing the real object interacting with the user's hand from the obtained image. The method may include identifying a pre-registered function which uses the recognized real object as an input tool. The method may include performing the identified pre-registered function. In an embodiment of the disclosure, a computer-readable recording medium having a program recorded thereon to cause a computer to perform the method of using a real object as an input tool of an electronic device. In an embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a camera for obtaining an image by photographing a real object in a real space. The electronic device may include memory storing at least one instruction. The electronic device may include at least one processor configured to execute the at least one instruction stored in the memory. The at least one processor may recognize the real object from the obtained image. The at least one processor may register the recognized real object as an input tool. The at least one processor may map an interaction between the registered real object and a user's hand to a predefined function. Brief Description of Drawings The present disclosure may be readily understood by combinations of the following detailed descriptions and the accompanying drawings, and reference numerals refer to structural elements. FIG. 1 is a conceptual diagram for describing an operation of using a real-world object (hereinafter, referred to as a real object) as an input tool of an electronic device, according to an embodiment of the present disclosure.FIG. 2 is a block diagram showing components of an electronic device, according to an embodiment of the present disclosure.FIG. 3 is a flowchart showing a method of using a real object as an input tool of an electronic device, according to an embodiment of the present disclosure.FIG. 4A is a conceptual diagram for describing an operation of generating or deleting a virtual object by using a real object, according to an embodiment of the present disclosure.FIG. 4B is a conceptual diagram for describing an operation of generating or deleting a virtual object by using a real object, according to an embodiment of the present disclosure.FIG. 5A is a conceptual diagram for describing an operation of generating or deleting a virtual object by using a real object, according to an embodiment of the present disclosure.FIG. 5B is a conceptual diagram for describing an operation of generating or deleting a virtual object by using a real object, according to an embodiment of the present disclosure.FIG. 6 is a conceptual diagram for describing an operation of determining a co