KR-102962931-B1 - AUGMENTED REALITY DEVICE OBTAINING DEPTH INFORMATION OF THE OBJECT AND METHOD FOR OPERATING THE SAME

KR102962931B1KR 102962931 B1KR102962931 B1KR 102962931B1KR-102962931-B1

Abstract

The present disclosure provides an augmented reality device for acquiring depth value information of an object located in a region of interest and a method for operating the same. An augmented reality device according to one embodiment of the present disclosure may calculate a Region of Interest Confidence Level, which indicates the degree to which at least one part area within the entire area of the real world is predicted to be a Region of Interest, based on at least one of the movement speed, acceleration, fixation time, number of fixations, and position of a point of interest acquired through an eye-tracking sensor; determine a region of interest based on the calculated Region of Interest Confidence Level; and set parameters for controlling the operation of a depth sensor to acquire depth value information of an object within the region of interest.

Inventors

윤상호
구본곤
최종철
윤정근
정영모

Assignees

삼성전자주식회사

Dates

Publication Date: 20260508
Application Date: 20210430

Claims (20)

In an augmented reality device for acquiring depth information of real-world objects, A gaze tracking sensor that acquires interest point information regarding the object by tracking the gaze of a user looking at the object; A depth sensor that obtains depth value information of the above object; Memory for storing one or more instructions; and A processor that executes one or more instructions stored in the above memory; Includes, The above processor is, Based on at least one piece of information among the movement speed, acceleration, fixation time, number of fixations, and position of the point of interest acquired through the eye-tracking sensor, a Region of Interest Confidence Level is calculated that indicates the degree to which at least one partial area within the entire area of the real world is predicted to be a Region of Interest. Based on the confidence level of the region of interest calculated above, the region of interest within the entire region is determined, and To obtain the depth value of an object within the above-mentioned region of interest, parameters for controlling the operation of the depth sensor are set, and An augmented reality device, wherein the processor calculates the reliability level of the region of interest according to an inverse relationship between at least one of the movement speed and acceleration of the point of interest.
In Article 1, The above processor is, Using the above eye tracking sensor, a first gaze vector representing the gaze direction of the user's left eye and a second gaze vector representing the gaze direction of the user's right eye are obtained, and Detecting a gaze point where the first gaze vector and the second gaze vector converge according to binocular disparity, An augmented reality device that acquires 2D position coordinate information of the detected gaze point.
In Article 2, The above processor is, An augmented reality device that calculates the region of interest confidence level based on at least one of the fixation time, fixation number, movement speed, acceleration, and position of the detected gaze point.
In Article 1, The above processor is, An augmented reality device that calculates the reliability level of the region of interest according to a proportional relationship with at least one of the fixed time and the number of fixed times of the point of interest.
delete
In Article 1, The above processor is, An augmented reality device that determines at least one partial region as the region of interest, wherein the confidence level of the region of interest calculated above is greater than or equal to a preset threshold.
In Article 1, A camera that acquires an image of the entire area by photographing the entire area of the real world; Includes more, An augmented reality device, wherein the processor adjusts the depth map resolution and frame rate of a portion corresponding to a determined region of interest among an image acquired through the camera to a higher value than that of other portions.
In Article 1, The depth sensor includes a light-emitting part configured to generate light and irradiate the generated light onto an object, and The above processor is an augmented reality device that resets the size of the region of interest so that a preset threshold amount of light and a preset threshold number of pattern lights are irradiated to the region of interest determined by the light-emitting unit.
In Article 8, The above-mentioned light-emitting unit includes a light source and an LC (Liquid Crystal)-based reflective mirror, and The above processor is, An augmented reality device that controls the power applied to the LC-based reflective mirror so that pattern light is irradiated to the region of interest by changing the arrangement of liquid crystal molecules included in the liquid crystal layer of the LC-based reflective mirror.
In Article 9, The depth sensor includes a light receiving sensor that receives light reflected from the object by being irradiated by the light source, and The light receiving sensor is divided into a plurality of sections, and the plurality of sections are sequentially controlled to turn on/off according to a preset clock signal. The above processor is, An augmented reality device that controls a light receiving sensor to sequentially switch at least one first section corresponding to the region of interest among the plurality of sections to an on state, and to keep at least one second section not corresponding to the region of interest in an off state.
In Article 10, A plurality of sections of the light receiving sensor include a plurality of SPADs (Single-Photon Avalanche Diodes), and The above processor is, An augmented reality device that disables at least one SPAD by blocking a bias voltage applied to at least one SPAD that does not include the region of interest among a plurality of SPADs included in at least one first section.
In a method of operation of an augmented reality device for acquiring depth value information of a real-world object, A step of obtaining point of interest information regarding the object by tracking the point of interest of a user who looks at or points to the object; A step of calculating an ROI Confidence Level indicating the degree to which at least one partial region within the entire area of the real world is predicted to be a Region of Interest, based on at least one piece of information among the movement speed, acceleration, fixed time, number of fixed times, and location of the above-mentioned point of interest; A step of determining a region of interest within the entire region based on the calculated region of interest confidence level; and A step of setting parameters for controlling the operation of a depth sensor of the augmented reality device used to obtain a depth value of an object within the aforementioned region of interest; Includes, The step of calculating the confidence level of the region of interest above is, A method comprising the step of calculating the reliability level of the region of interest according to an inverse relationship between at least one of the movement speed and acceleration of the point of interest.
In Article 12, The step of obtaining information regarding the above points of interest is, A step of obtaining a first gaze vector representing the gaze direction of the user's left eye and a second gaze vector representing the gaze direction of the user's right eye using an eye tracking sensor; A step of detecting a gaze point where the first gaze vector and the second gaze vector converge according to binocular disparity; and A step of obtaining 2D position coordinate information of the detected gaze point; A method including
In Article 13, The step of calculating the confidence level of the region of interest above is, A method for calculating the confidence level of the region of interest based on at least one of the fixation time, fixation number, movement speed, and acceleration of the detected gaze point, which remains on the same region or same object.
In Article 12, The step of determining the above region of interest is, A method for determining at least one partial region as the region of interest, wherein the confidence level of the region of interest calculated above is greater than or equal to a preset threshold.
In Article 12, The depth sensor includes a light-emitting part configured to generate light and irradiate the generated light onto an object, and The step of setting the parameters of the depth sensor above is, A method for resetting the size of a region of interest such that a predetermined threshold amount of light and a predetermined threshold number of pattern lights are irradiated to the region of interest determined by the light-emitting unit.
In Article 16, The above-mentioned light-emitting unit includes a light source and an LC-based reflective mirror, and The step of setting the parameters of the depth sensor above is, A method comprising the step of adjusting the power applied to the LC-based reflective mirror so that pattern light is irradiated to the region of interest by changing the arrangement of liquid crystal molecules included in the liquid crystal layer of the LC-based reflective mirror.
In Article 17, The depth sensor includes a light receiving sensor that receives light reflected from the object by being irradiated by the light source, and The light receiving sensor is divided into a plurality of sections, and the plurality of sections are sequentially controlled to turn on/off according to a preset clock signal. The step of setting the parameters of the depth sensor above is, A method of sequentially switching at least one first section corresponding to the region of interest to an on state among the plurality of sections above, and maintaining at least one second section not corresponding to the region of interest in an off state.
In Article 18, The light receiving sensor described above comprises, wherein each of the plurality of sections includes a plurality of SPADs (Single-Photon Avalanche Diodes), A method comprising the step of deactivating at least one SPAD by blocking a bias voltage applied to at least one SPAD that does not include the region of interest among a plurality of SPADs included in at least one first section.
A computer-readable recording medium having a program for executing the method of claim 12 on a computer.

Description

Augmented Reality Device Obtaining Depth Information of an Object and Method for Operating the Same The present disclosure relates to a device for acquiring depth information of an object and a method of operating the same, and specifically, to a device for setting parameters of a depth sensor to acquire depth information regarding a region of interest that includes an object that a user looks at or points to within the entire area included in the field of view (FOV), and a method of operating the same. Augmented Reality is a technology that overlays virtual images onto the physical environment of the real world or real-world objects to display them together. Augmented Reality devices utilizing this technology (e.g., smart glasses) are being used effectively in daily life for tasks such as information retrieval, navigation, and photography. In particular, smart glasses are also worn as fashion items and are primarily used for outdoor activities. Recently, devices including depth sensors that acquire depth information are widely used to express the spatial sense of objects included in real space composed of three-dimensional volumetric space. Conventional depth information acquisition technology using depth sensors performs depth sensing for all objects included in the entire area within the field of view (FOV) and acquires depth value information without considering the user's area of interest. However, due to recent limitations such as device miniaturization and battery capacity for portability, performing depth sensing across the entire field of view increases computational load and consequently power consumption. Furthermore, since a user's gaze at an object or hand gestures pointing at it are continuous and change in real time, performing depth sensing across the entire area and then re-performing it for areas that change based on gaze or hand gestures results in a problem of unnecessary power consumption. FIG. 1 is a conceptual diagram for explaining a method of operation of an augmented reality device according to one embodiment of the present disclosure. FIG. 2 is a block diagram illustrating the components of an augmented reality device according to one embodiment of the present disclosure. FIG. 3 is a flowchart illustrating a method of operation of an augmented reality device according to one embodiment of the present disclosure. FIG. 4 is a diagram illustrating a method in which an augmented reality device according to one embodiment of the present disclosure determines a region of interest based on the user's gaze direction. FIG. 5a is a drawing illustrating an eye-tracking sensor, which is a component of an augmented reality device according to one embodiment of the present disclosure. FIG. 5b is a drawing illustrating an eye-tracking sensor, which is a component of an augmented reality device according to one embodiment of the present disclosure. Fig. 5c is a diagram illustrating a three-dimensional eye model of the user's gaze direction. FIG. 6a is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to calculate a gaze point from information about the direction of gaze measured by an eye-tracking sensor. FIG. 6b is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to calculate a gaze point from information about the direction of gaze measured by an eye-tracking sensor. FIG. 7 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to calculate a region of interest confidence level based on a gaze point according to the user's gaze direction. FIG. 8 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to calculate a region of interest confidence level based on the position of a user's hand. FIG. 9 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to calculate a region of interest reliability level based on the position of an input controller. FIG. 10 is a drawing for explaining the structure and operation method of a depth sensor according to one embodiment of the present disclosure. FIG. 11 is a diagram illustrating an embodiment in which an augmented reality device of the present disclosure resets the size of a region of interest based on the amount and number of pattern lights irradiated from the light-emitting part of a depth sensor. FIG. 12 is a diagram illustrating an embodiment in which an augmented reality device of the present disclosure controls the refractive power of an LC-based reflective mirror, which is a component of a depth sensor. FIG. 13 is a diagram illustrating an embodiment in which an augmented reality device of the present disclosure controls the operation of a light receiving sensor, which is a component of a depth sensor, based on a re