US-12625567-B2 - User interface device, user interface system, and recording medium

Abstract

A location of an indicator in a camera image and a shape of the indicator are accurately detected. A user interface device configured to specify an indication by identifying a shape of an indicator based on a camera image obtained by capturing the indicator having the shape that changes for the indication includes a detector configured to search an image obtained by binarizing the camera image in at least two different diagonal directions to detect an end of an object included in the camera image, an image cropping unit configured to crop a partial image from the camera image based on the detected end, a selector configured to select a partial image indicating a shape before the indication and a partial image indicating a shape of the indication by the indicator when the partial image is a partial image obtained by capturing the indicator as the object, and an information generator configured to generate information indicating whether the indicator has the shape before the indication or the shape of the indication.

Inventors

• Ryuichi Kojima

Assignees

• DEEP INSIGHT INC.

Dates

Publication Date

20260512

Application Date

20220502

Claims (19)

1. 1 . A user interface device configured to specify an indication by identifying a shape of an indicator based on a camera image obtained by capturing the indicator having the shape that changes for the indication, the user interface device comprising: a detector configured to search an image obtained by binarizing the camera image in at least two different diagonal directions to detect one end of an object included in the camera image; an image cropping unit configured to crop a partial image corresponding to a detected position of the end of the object and a size of a region including the end of the object from the camera image based on the detected end; an extraction selection configured to extract a partial image indicating a shape before the indication which is a pointer in a non-selected state and a partial image indicating a shape of the indication which is a pointer in a selected state from partial images; and an information generator configured to generate information indicating whether the indicator has the shape before the indication which is the pointer in a non-selected state or the shape of the indication which is the pointer in a selected state.
2. 2 . The user interface device according to claim 1 , comprising a shape recognition unit configured to classify the partial image by deep learning, wherein the extraction selection extracts a partial image in which the indicator indicates the shape before the indication and a partial image in which the indicator indicates the shape of the indication from the classified partial image.
3. 3 . The user interface device according to claim 1 , wherein: the indicator is a thumb and an index finger of a user, and the shape before the indication is a state in which the thumb and the index finger are open, and the shape of the indication is a state in which the thumb and the index finger are in contact.
4. 4 . The user interface device according to claim 1 , wherein: the information generator includes a coordinate storage configured to store coordinates indicating a position of the indicator, and a moving average value calculator configured to read a history of the coordinates stored in the coordinate storage, obtain a moving average value, correct the input coordinates using the moving average value, and output the corrected coordinates in response to input of the coordinates indicating the position of the indicator, and the coordinates indicating the position of the indicator are output from the extraction selection.
5. 5 . The user interface device according to claim 1 , wherein: the information generator includes a coordinate storage configured to store coordinates indicating a position of the indicator, a coordinate lock controller configured to detect a change in the shape of the indicator and outputs a lock control signal for locking the position of the indicator when the shape changes based on the generated information indicating whether the indicator has the shape before the indication or the shape of the indication, and a coordinate selector configured to output the input coordinates indicating the position of the indicator when the lock control signal is not input, and to read coordinates of the indicator immediately before locking stored in the coordinate storage, maintain the coordinates of the indicator immediately before a shape changes, and output the coordinates of the indicator when the lock control signal is input, and the coordinates indicating the position of the indicator are output from the coordinate selector.
6. 6 . The user interface device according to claim 5 , comprising a coordinate change amount detector configured to detect a coordinate change amount by reading the coordinates indicating the position of the indicator immediately before locking from the coordinate storage in response to input of the coordinates indicating the position of the indicator, wherein: in a case of detecting the change in the shape of the indicator, when a change amount from the immediately previous coordinates does not exceed an input predetermined change amount, the coordinate lock controller outputs the lock control signal, and when the change amount exceeds the input predetermined change amount, the coordinate lock controller does not output the lock control signal and outputs a lock release control signal when the change amount from the immediately previous coordinates exceeds the input predetermined change amount after the lock control signal is output, and in a case of receiving the lock release control signal, the coordinate selector releases maintaining of the coordinates of the indicator and outputs the input coordinates indicating the position of the indicator.
7. 7 . The user interface device according to claim 5 , wherein: the coordinate lock controller detects a change in the shape of the indicator and outputs a lock release control signal when a predetermined time elapses after the lock control signal is output, and in a case of receiving the lock release control signal, the coordinate selector releases maintaining of the coordinates of the indicator and outputs the input coordinates indicating the position of the indicator.
8. 8 . The user interface device according to claim 6 , wherein the coordinate lock controller detects a change in the shape of the indicator and rewrites the input predetermined change amount to a value smaller than the predetermined change amount when a predetermined time elapses after the lock control signal is output.
9. 9 . The user interface device according to claim 1 , wherein the information generator includes a storage configured to store information indicating whether the generated indicator is in a state of the shape before the indication or in a state of the shape of the indication, a continuous state detector configured to read the state of the pointer from the storage, and outputs a switching control signal in the state of the indicator when the state of the indicator changes and the changed state of the indicator is detected a predetermined number of times in succession, a state selector configured to switch a state output before input to a state indicated by input information in response to receiving the switching control signal when information indicating whether the generated indicator is in the state of the shape before the indication or a state of the shape of the indication, and a state indicated by the input information is different from a state output before the input, and a selected state storage configured to store the state output before the input.
10. 10 . The user interface device according to claim 1 , comprising: a determination unit configured to determine whether or not the position of the indicator is outside a region of the camera image; and a notification unit configured to notify the user when the position of the indicator is outside the region of the camera image.
11. 11 . A user interface system comprising: the user interface device according to claim 1 ; an imaging device configured to capture an indicator having a shape that changes for an indication; and a display device configured to display a display screen operated based on information indicating whether the indicator output from the user interface device has a shape before the indication or a shape of the indication.
12. 12 . A computer readable non-transitory recording medium recording a user interface program configured to cause a computer to execute: a detection function of performing a search in at least two different diagonal directions to detect one end of an object included in a camera image; an image cropping function of cropping a partial image corresponding to a detected position of the end of the object and a size of a region including the end of the object from the camera image based on the detected end; an extraction function of extracting a partial image indicating a shape before an indication which is a pointer in a non-selected state and a partial image indicating a shape of the indication which is a pointer in a selected state from partial images; and an information generation function of generating information indicating whether the indicator has the shape before the indication which is the pointer in a non-selected state or the shape of the indication which is the pointer in a selected state.
13. 13 . A user interface device configured to specify an indication by identifying a shape of an indicator based on a camera image obtained by capturing the indicator having the shape that changes for the indication, the user interface device comprising: a detector configured to search an image obtained by binarizing the camera image in at least two different diagonal directions to detect an end of an object included in the camera image; an image cropping unit configured to crop a partial image from the camera image based on the detected end; a selector configured to select a partial image indicating a shape before the indication and a partial image indicating a shape of the indication by the indicator when the partial image is a partial image obtained by capturing the indicator as the object; and an information generator configured to generate information indicating whether the indicator has the shape before the indication or the shape of the indication, wherein: (a) the information generator includes a coordinate storage configured to store coordinates indicating a position of the indicator, and a moving average value calculator configured to read a history of the coordinates stored in the coordinate storage, obtain a moving average value, correct the input coordinates using the moving average value, and output the corrected coordinates in response to input of the coordinates indicating the position of the indicator, and the coordinates indicating the position of the indicator are output from the selector; or (b) the information generator includes a coordinate storage configured to store coordinates indicating a position of the indicator, a coordinate lock controller configured to detect a change in the shape of the indicator and outputs a lock control signal for locking the position of the indicator when the shape changes based on the generated information indicating whether the indicator has the shape before the indication or the shape of the indication, and a coordinate selector configured to output the input coordinates indicating the position of the indicator when the lock control signal is not input, and to read coordinates of the indicator immediately before locking stored in the coordinate storage, maintain the coordinates of the indicator immediately before a shape changes, and output the coordinates of the indicator when the lock control signal is input, and the coordinates indicating the position of the indicator are output from the selector; or (c) wherein the information generator includes a storage configured to store information indicating whether the generated indicator is in a state of the shape before the indication or in a state of the shape of the indication, a continuous state detector configured to read the state of the pointer from the storage, and outputs a switching control signal in the state of the indicator when the state of the indicator changes and the changed state of the indicator is detected a predetermined number of times in succession, a state selector configured to switch a state output before input to a state indicated by input information in response to receiving the switching control signal when information indicating whether the generated indicator is in the state of the shape before the indication or a state of the shape of the indication, and a state indicated by the input information is different from a state output before the input, and a selected state storage configured to store the state output before the input.
14. 14 . The user interface device according to claim 13 , wherein: the information generator includes a coordinate storage configured to store coordinates indicating a position of the indicator, and a moving average value calculator configured to read a history of the coordinates stored in the coordinate storage, obtain a moving average value, correct the input coordinates using the moving average value, and output the corrected coordinates in response to input of the coordinates indicating the position of the indicator, and the coordinates indicating the position of the indicator are output from the selector.
15. 15 . The user interface device according to claim 13 , wherein: the information generator includes a coordinate storage configured to store coordinates indicating a position of the indicator, a coordinate lock controller configured to detect a change in the shape of the indicator and outputs a lock control signal for locking the position of the indicator when the shape changes based on the generated information indicating whether the indicator has the shape before the indication or the shape of the indication, and a coordinate selector configured to output the input coordinates indicating the position of the indicator when the lock control signal is not input, and to read coordinates of the indicator immediately before locking stored in the coordinate storage, maintain the coordinates of the indicator immediately before a shape changes, and output the coordinates of the indicator when the lock control signal is input, and the coordinates indicating the position of the indicator are output from the selector.
16. 16 . The user interface device according to claim 15 , comprising a coordinate change amount detector configured to detect a coordinate change amount by reading the coordinates indicating the position of the indicator immediately before locking from the coordinate storage in response to input of the coordinates indicating the position of the indicator, wherein: in a case of detecting the change in the shape of the indicator, when a change amount from the immediately previous coordinates does not exceed an input predetermined change amount, the coordinate lock controller outputs the lock control signal, and when the change amount exceeds the input predetermined change amount, the coordinate lock controller does not output the lock control signal and outputs a lock release control signal when the change amount from the immediately previous coordinates exceeds the input predetermined change amount after the lock control signal is output, and in a case of receiving the lock release control signal, the coordinate selector releases maintaining of the coordinates of the indicator and outputs the input coordinates indicating the position of the indicator.
17. 17 . The user interface device according to claim 15 , wherein: the coordinate lock controller detects a change in the shape of the indicator and outputs a lock release control signal when a predetermined time elapses after the lock control signal is output, and in a case of receiving the lock release control signal, the coordinate selector releases maintaining of the coordinates of the indicator and outputs the input coordinates indicating the position of the indicator.
18. 18 . The user interface device according to claim 16 , wherein the coordinate lock controller detects a change in the shape of the indicator and rewrites the input predetermined change amount to a value smaller than the predetermined change amount when a predetermined time elapses after the lock control signal is output.
19. 19 . The user interface device according to claim 13 , wherein: the information generator includes a storage configured to store information indicating whether the generated indicator is in a state of the shape before the indication or in a state of the shape of the indication, a continuous state detector configured to read the state of the pointer from the storage, and outputs a switching control signal in the state of the indicator when the state of the indicator changes and the changed state of the indicator is detected a predetermined number of times in succession, a state selector configured to switch a state output before input to a state indicated by input information in response to receiving the switching control signal when information indicating whether the generated indicator is in the state of the shape before the indication or a state of the shape of the indication, and a state indicated by the input information is different from a state output before the input, and a selected state storage configured to store the state output before the input.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is the national phase under 35 U.S.C. § 371 of PCT International Application No. PCT/JP2022/019519 which has an International filing date of May 2, 2022 and designated the United States of America. BACKGROUND ART A user interface device that performs input to a computer by movement of a fingertip of a user is described in, for example, Japanese Patent Laid-Open Publication No. 2011-133942. Japanese Patent Laid-Open Publication No. 2011-133942 describes a user interface device that can change a screen displayed on a touch panel in response to movement of a fingertip of a user even when the fingertip is not touching the touch panel. Specifically, Japanese Patent Laid-Open Publication No. 2011-133942 describes that a user interface device includes an image output means capable of outputting a display image to be displayed on a display, an imaging means for capturing a peripheral image of the periphery of the display, an analysis means for analyzing a state of a finger of the user based on the captured peripheral image, an estimation means for estimating operation intention of the user based on the analyzed state of the finger, and a control means for controlling the image output means to change the display image according to the estimated operation intention before a fingertip of the user touches a surface of the display. SUMMARY In a user interface device that exchanges information with another device based on movement of an indicator, such as a finger, recognized from a camera image, it is necessary to accurately detect a location of the indicator in the camera image and a shape of the indicator. An object of the disclosure is to provide a user interface device, a user interface system, and a recording medium capable of accurately detecting a location of an indicator in a camera image and a shape of the indicator. (1) A first aspect of the disclosure is a user interface device configured to specify an indication by identifying a shape of an indicator based on a camera image obtained by capturing the indicator having the shape that changes for the indication, the user interface device including a detector configured to search an image obtained by binarizing the camera image in at least two different diagonal directions to detect an end of an object included in the camera image, an image cropping unit configured to crop a partial image from the camera image based on the detected end, a selector configured to select a partial image indicating a shape before the indication and a partial image indicating a shape of the indication by the indicator when the partial image is a partial image obtained by capturing the indicator as the object, and an information generator configured to generate information indicating whether the indicator has the shape before the indication or the shape of the indication.(2) A second aspect of the disclosure is a user interface system including the user interface device of (1), an imaging device configured to capture an indicator having a shape that changes for an indication, and a display device configured to display a display screen operated based on information indicating whether the indicator output from the user interface device has a shape before the indication or a shape of the indication.(3) A third aspect of the disclosure is a computer readable non-transitory recording medium recording a user interface program configured to cause a computer to execute a detection function of performing a search in at least two different diagonal directions to detect an end of an object included in a camera image, an image cropping function of cropping a partial image from the camera image based on the detected end, a selection function of selecting a partial image indicating a shape before an indication and a partial image indicating a shape of the indication by an indicator when the partial image is a partial image obtained by capturing the indicator having a shape that changes for the indication as the object, and an information generation function of generating information indicating whether the indicator has the shape before the indication or the shape of the indication. According to the disclosure, it is possible to accurately detect a location of an indicator in a camera image and a shape of the indicator. The above and further objects and features of the invention will more fully be apparent from the following detailed description with accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a block diagram illustrating a user interface system of an embodiment of the disclosure. FIG. 2 is an explanatory diagram illustrating an action of selecting a numeric key by pinching using a thumb and an index finger. FIG. 3 is a block diagram illustrating a configuration of a user interface device. FIG. 4 is a diagram illustrating a camera image including a right hand. FIG. 5 is a diagram describing a method of searching for a fingert