US-12625562-B2 - Display device recognizing user gesture through sensor and controlling method thereof

US12625562B2US 12625562 B2US12625562 B2US 12625562B2US-12625562-B2

Abstract

A display device is provided. The display device includes: a display, a sensor and at least one processor, comprising processing circuitry, individually and/or collectively, configured to: control the display to display an indicator, recognize a user's body in an image received through the sensor in real time, identify a first area of the body as a reference point, and identify a second area of the body as an auxiliary point, identify a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image, and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point.

Inventors

Sunwoong JOO
Jongin LEE
Yongdeok Kim

Assignees

SAMSUNG ELECTRONICS CO., LTD.

Dates

Publication Date: 20260512
Application Date: 20250131
Priority Date: 20220801

Claims (17)

1 . A display device comprising: a display; a sensor; and at least one processor, comprising processing circuitry, individually and/or collectively, configured to: control the display to display an indicator; recognize a user's body in an image received through the sensor in real time; identify a first area of the user's body as a reference point, and identify a second area of the user's body as an auxiliary point; identify a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image; and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point is a relative change in the coordinates of the auxiliary point relative to the reference point, and wherein the at least one processor, individually and/or collectively, is configured to: apply a first weight to the change in the coordinates of the reference point, and apply a second weight to the change in the coordinates of the auxiliary point; and move the indicator by using the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied; and wherein the first weight is greater than the second weight.
2 . The display device as claimed in claim 1 , wherein at least one processor, individually and/or collectively, is configured to: based on the change in the coordinates of the reference point being less than a threshold value for a specified period of time, adjust the first weight downward.
3 . The display device as claimed in claim 1 , wherein at least one processor, individually and/or collectively, is configured to: based on the change in the coordinates of the auxiliary point being equal to or greater than a threshold value for a specified period of time, adjust the second weight upward.
4 . The display device as claimed in claim 1 , further comprising: memory storing weight information for each user position, wherein at least one processor, individually and/or collectively, is configured to: identify a position corresponding to the user's body; identify weight information corresponding to the identified position; and apply the first weight to the change in the coordinates of the reference point and apply the second weight to the change in the coordinates of the auxiliary point based on the identified weight information.
5 . The display device as claimed in claim 4 , wherein at least one processor, individually and/or collectively, is configured to: based on the user's position being identified as a first position according to the image, apply a first weight included in first weight information corresponding to the first position to the change in the coordinates of the reference point and apply a second weight included in the first weight information to the change in the coordinates of the auxiliary point; and based on the user's position being changed from the first position to a second position according to the image, apply a first weight included in second weight information corresponding to the second position to the change in the coordinates of the reference point, and apply a second weight included in the second weight information to the change in the coordinates of the auxiliary point.
6 . The display device as claimed in claim 1 , wherein at least one processor, individually and/or collectively, is configured to: identify a third area in the image; based on a specified operation being detected in the third area, obtain a selection command for the indicator; and perform an operation corresponding to the selection command.
7 . The display device as claimed in claim 6 , wherein at least one processor, individually and/or collectively, is configured to: identify an area where each of the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point is less than a threshold value while the specified operation is performed as the third area, or identify a specified area within the user's body as the third area.
8 . The display device as claimed in claim 1 , wherein the first area includes the user's wrist; and wherein the second area includes one of the user's fingers.
9 . A method of controlling a display device, the method comprising; displaying an indicator; recognizing a user's body in an image received in real time; identifying a first area of the user's body as a reference point, and identifying a second area of the user's body as an auxiliary point; identifying a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image; and moving the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point is a relative change in the coordinates of the auxiliary point relative to the reference point, wherein the moving the indicator comprises: applying a first weight to the change in the coordinates of the reference point, and applying a second weight to the change in the coordinates of the auxiliary point; and moving the indicator by using the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied; and wherein the first weight is greater than the second weight.
10 . The method as claimed in claim 9 , further comprising: based on the change in the coordinates of the reference point being less than a threshold value for a specified period of time, adjusting the first weight downward.
11 . The method as claimed in claim 9 , further comprising: based on the change in the coordinates of the auxiliary point being equal to or greater than a threshold value for a specified period of time, adjusting the second weight upward.
12 . The method as claimed in claim 9 , wherein the moving the indicator comprises: identifying a position corresponding to the user's body; identifying weight information corresponding to the identified position; and applying the first weight to the change in the coordinates of the reference point and applying the second weight to the change in the coordinates of the auxiliary point based on the identified weight information.
13 . The method as claimed in claim 12 , wherein the moving the indicator comprises: based on the user's position being identified as a first position according to the image, applying a first weight included in first weight information corresponding to the first position to the change in the coordinates of the reference point and applying a second weight included in the first weight information to the change in the coordinates of the auxiliary point; and based on the user's position being changed from the first position to a second position according to the image, applying a first weight included in second weight information corresponding to the second position to the change in the coordinates of the reference point, and applying a second weight included in the second weight information to the change in the coordinates of the auxiliary point.
14 . A non-transitory computer-readable recording medium storing computer instructions that when executed by at least one processor, comprising processing circuitry, of a display device, individually and/or collectively, cause the display device to: display an indicator; recognize a user's body in an image received in real time; identify a first area of the user's body as a reference point, and identifying a second area of the user's body as an auxiliary point; identify a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image; and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point is a relative change in the coordinates of the auxiliary point relative to the reference point, wherein the computer instructions further cause the display device to: apply a first weight to the change in the coordinates of the reference point, and applying a second weight to the change in the coordinates of the auxiliary point; and move the indicator by using the change in coordinates where the first weight is applied and the change in coordinates where the second weight is applied; and wherein the first weight is greater than the second weight.
15 . The non-transitory computer-readable recording medium of claim 14 , wherein the computer instructions further cause the display device to: based on the change in the coordinates of the reference point being less than a threshold value for a specified period of time, adjust the first weight downward.
16 . The non-transitory computer-readable recording medium of claim 14 , wherein the computer instructions further cause the display device to: based on the change in the coordinates of the auxiliary point being equal to or greater than a threshold value for a specified period of time, adjust the second weight upward.
17 . The non-transitory computer-readable recording medium of claim 14 , wherein the computer instructions further cause the display device to: identify a position corresponding to the user's body; identify weight information corresponding to the identified position; and apply the first weight to the change in the coordinates of the reference point and applying the second weight to the change in the coordinates of the auxiliary point based on the identified weight information.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of International Application No. PCT/KR2023/009403 designating the United States, filed on Jul. 4, 2023, in the Korean Intellectual Property Receiving Office and claiming priority to Korean Patent Application No. 10-2022-0095714, filed on Aug. 1, 2022, in the Korean Intellectual Property Office, the disclosures of each of which are incorporated by reference herein in their entireties. BACKGROUND Field The disclosure relates to a display device that recognizes a user gesture via a sensor and a controlling method thereof, and for example, to a display device that recognizes a user gesture and moves and displays an indicator and a controlling method thereof. Description of Related Art Recently, with technological advances in the field of display devices, various types of display devices have been developed and popularized. Various types of user interfaces for controlling display devices have also been developed and popularized. In particular, research on the so-called Natural User Interface (NUI) that detects natural human body movements and controls a display device using them rather than controlling the display device using peripheral devices such as a keyboard, a mouse, or a remote controller has been actively conducted. When controlling a device using body movements, the challenge is to detect the body movements with high accuracy so that fine control can be achieved while ignoring unnecessary body movements to match the user's intent. When controlling a display device by detecting the user's body movements and moving a cursor, etc. according to the detected body movements, there is a growing need for a device controlling method that accurately reflects even the slightest movements of the cursor. SUMMARY A display device according to an example embodiment includes: a display, a sensor and at least one processor, comprising processing circuitry, individually and/or collectively, configured to: control the display to display an indicator, recognize a user's body in an image received through the sensor in real time, identify a first area of the body as a reference point, identify a second area of the body as an auxiliary point, identify a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image, and move the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point corresponds to the relative coordinate change of the auxiliary point with reference to the reference point. A method of controlling a display device according to an example embodiment includes: displaying an indicator, recognizing a user's body in an image received in real time, identifying a first area of the body as a reference point, and identifying a second area of the body as an auxiliary point, identifying a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image, and moving the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point corresponds to the relative coordinate change of the auxiliary point with reference to the reference point. In a non-transitory computer-readable recording medium including a program that when executed causes a display device to perform a method of controlling the display device according to an example embodiment, the method of controlling the display device including: displaying an indicator, recognizing a user's body in an image received in real time, identifying a first area of the body as a reference point, and identifying a second area of the body as an auxiliary point, identifying a change in coordinates of the reference point and a change in coordinates of the auxiliary point based on a plurality of image frames included in the image, moving the indicator based on the change in the coordinates of the reference point and the change in the coordinates of the auxiliary point, wherein the change in the coordinates of the auxiliary point corresponds to the relative coordinate change of the auxiliary point with reference to the reference point. BRIEF DESCRIPTION OF THE DRAWINGS The above and other aspects, features and advantages of certain embodiments of the present disclosure will be more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which: FIG. 1 is a diagram illustrating an example display device according to various embodiments; FIG. 2 is a block diagram illustrating an example configuration of a display device according to various embodiments; FIG. 3 is a diagram illustrating movement of an indicator in response to a