Search

EP-4254138-B1 - METHOD FOR CONTROLLING AUGMENTED REALITY DEVICE, AND AUGMENTED REALITY DEVICE PERFORMING SAME

EP4254138B1EP 4254138 B1EP4254138 B1EP 4254138B1EP-4254138-B1

Inventors

  • HAN, Insun
  • KWON, Doil
  • Kim, Jiin
  • CHUN, JAEMIN
  • HAN, Minseok

Dates

Publication Date
20260506
Application Date
20211224

Claims (15)

  1. An augmented reality, AR, device comprising: an eye tracking sensor (140) configured to obtain gaze information of a user; a communication interface module (150); a camera (130); a display engine unit (170); a display (160); a memory (110) storing one or more instructions; and a processor (120) configured to execute the one or more instructions, wherein the processor (120) is configured to: transmit gaze coordinates corresponding to a real-world object based on the gaze information to an image processing device that displays (160) the real-world object, through the communication interface module (150), recognize a gesture to call a virtual image of the real-world object specified in the image processing device according to screen coordinates corresponding to the gaze coordinates, based on an image captured through the camera (130), receive object information of the specified real-world object from the image processing device in response to transmitting an event notification corresponding to the recognized gesture to the image processing device, through the communication interface module (150), and characterized in that the processor (120) is further configured to: control the display engine unit (170) to provide the virtual image corresponding to the object information among prepared virtual images to a depth layer of the display (160) corresponding to a type of the specified real-world object.
  2. The AR device (100) of claim 1, wherein the processor (120) is further configured to, by executing the one or more instructions, provide virtual images of specified real-world objects to different depth layers of the display (160) when types of the specified real-world objects are different from each other.
  3. The AR device (100) of claim 1, wherein the processor (120) is further configured to, by executing the one or more instructions, provide a first virtual image of a first-type real-world object to a first depth layer of the display (160) and provide a second virtual image of a second-type real-world object to a second depth layer of the display (160).
  4. The AR device (100) of claim 3, wherein the first depth layer of the display (160) displays (160) a pop-up of a virtual image so as to be observed at a closer position than the second depth layer of the display (160).
  5. The AR device (100) of claim 1, wherein the processor (120) is further configured to, by executing the one or more instructions, provide a virtual image of a setting user interface to a first depth layer of the display (160) when the specified real-world object is the setting user interface, and provide a virtual image of media content to a second depth layer of the display (160) when the specified real-world object is the media content.
  6. The AR device (100) of claim 1, wherein the object information comprises identification information of the specified real-world object and data information, and the processor (120) is further configured to, by executing the one or more instructions, control the display engine unit (170) to reflect the data information in a virtual image corresponding to the identification information among the prepared virtual images and provide the virtual image to the depth layer of the display (160).
  7. The AR device (100) of claim 1, wherein the prepared virtual images are received from a server (200) through the communication interface module (150) when the AR device (100) is communication-connected to the server (200) that relays between the AR device (100) and the image processing device.
  8. The AR device (100) of claim 1, wherein the processor (120) is further configured to, by executing the one or more instructions, recognize a gesture to call the virtual image when a certain hand pose is detected in a hovered state of the specified real-world object, based on the image captured through the camera (130).
  9. The AR device (100) of claim 1, wherein the processor (120) is further configured to, by executing the one or more instructions, recognize a gesture to manipulate a pop-up of the provided virtual image based on the image captured through the camera (130) and transmit object information changed by the manipulation to the image processing device through the communication interface module (150).
  10. The AR device (100) of claim 1, wherein the display (160) comprises a plurality of depth layers where a pop-up of the virtual image is observed at different positions.
  11. A method of controlling an augmented reality, AR, device, the method comprising: transmitting gaze coordinates corresponding to a real-world object to an image processing device that displays (160) the real-world object; recognizing a gesture to call a virtual image of the real-world object specified in the image processing device according to screen coordinates corresponding to the gaze coordinates; receiving object information of the specified real-world object from the image processing device in response to transmitting an event notification corresponding to the recognized gesture to the image processing device; and characterized by providing the virtual image corresponding to the object information among prepared virtual images to a depth layer corresponding to a type of the specified real-world object.
  12. The method of claim 11, wherein the providing comprises providing virtual images of specified real-world objects to different depth layers when types of the specified real-world objects are different from each other.
  13. The method of claim 11, wherein the providing comprises providing a virtual image of a setting user interface to a first depth layer when the specified real-world object is the setting user interface and providing a virtual image of media content to a second depth layer when the specified real-world object is the media content.
  14. The method of claim 11, wherein the object information comprises identification information of the specified real-world object and data information, and the providing comprises reflecting the data information in a virtual image corresponding to the identification information among the prepared virtual images and providing the virtual image to the depth layer.
  15. A computer-readable recording medium having recorded thereon a program to be executed by a computer, the computer-readable recording medium comprising: instructions to transmit gaze coordinates corresponding to a real-world object to an image processing device that displays (160) the real-world object; instructions to recognize a gesture to call a virtual image of the real-world object specified in the image processing device according to screen coordinates corresponding to the gaze coordinates; instructions to receive object information of the specified real-world object from the image processing device in response to transmitting an event notification corresponding to the recognized gesture to the image processing device; and characterized by instructions to provide a virtual image corresponding to the object information among prepared virtual images to a depth layer corresponding to a type of the specified real-world object.

Description

Technical Field The present disclosure relates to a method of controlling an augmented reality (AR) device, and the AR device that performs the method. Background Art Augmented reality (AR) is a technology that projects a virtual image onto a physical environment space or a real-world object in the real world and displays the same as a single image. An AR device, while worn on a user's face or head, allows a user to see a real scene and a virtual image together through a pop-up of the virtual image in the form of a projection, placed in front of the user's eyes. US 2020/371673 A1 discloses a system allowing users to efficiently modify an arrangement of a user interface of a communication session by the use of an eye gaze gesture. An eye gaze gesture input can be utilized to focus on particular aspects of shared content. In addition, an eye gaze gesture can be utilized to configure an arrangement of a user interface displaying multiple streams of shared content of a communication session. A focused view of shared content and customized user interface layouts can be shared with specific individuals based on roles and or permissions. In addition, also unique user interface controls can be selected and displayed based on an eye gaze gesture. In one illustrative example, a specific set of functionality can be made available to a user based on a type of an object that is selected using an eye gaze gesture. Disclosure Technical Problem Provided are an AR device that provides a virtual image of a real-world object displayed on an image processing device through interaction with the image processing device, and a method of controlling the AR device. Technical Solution According to a first aspect, a method of controlling an augmented reality (AR) device includes transmitting gaze coordinates corresponding to a real-world object to an image processing device that displays the real-world object, recognizing a gesture to call a virtual image of the real-world object specified in the image processing device according to screen coordinates corresponding to the gaze coordinates, receiving object information of the specified real-world object from the image processing device in response to transmitting an event notification corresponding to the recognized gesture to the image processing device, and providing a virtual image corresponding to the object information among prepared virtual images to a depth layer corresponding to a type of the specified real-world object. According to a second aspect, a computer-readable recording medium having recorded thereon a program to be executed by a computer includes instructions to transmit gaze coordinates corresponding to a real-world object to an image processing device that displays the real-world object, instructions to recognize a gesture to call a virtual image of the real-world object specified in the image processing device according to screen coordinates corresponding to the gaze coordinates, instructions to receive object information of the specified real-world object from the image processing device in response to transmitting an event notification corresponding to the recognized gesture to the image processing device, and instructions to provide a virtual image corresponding to the object information among prepared virtual images to a depth layer corresponding to a type of the specified real-world object. According to a third aspect, an AR device includes an eye tracking sensor configured to obtain gaze information of a user, a communication interface module, a camera, a display engine unit, a display, a memory storing one or more instructions, and a processor configured to execute the one or more instructions, wherein the processor is configured to transmit gaze coordinates corresponding to a real-world object based on the gaze information to an image processing device that displays the real-world object, through the communication interface module, recognize a gesture to call a virtual image of the real-world object specified in the image processing device according to screen coordinates corresponding to the gaze coordinates, based on an image captured through the camera, receive object information of the specified real-world object from the image processing device in response to transmitting an event notification corresponding to the recognized gesture to the image processing device, through the communication interface module, and control the display engine unit to provide a virtual image corresponding to the object information among prepared virtual images to a depth layer of the display corresponding to a type of the specified real-world object. Description of Drawings FIG. 1 is a view for describing an environment where an augmented reality (AR) device interacts with an image processing device, according to an embodiment of the present disclosure.FIG. 2 is a view for describing a process of providing, by an AR device, a virtual image of a real-world object displayed o