Search

US-12625558-B2 - Method and apparatus for user-defined gesture profile generation and gesture recognition, based on repeated motion of user

US12625558B2US 12625558 B2US12625558 B2US 12625558B2US-12625558-B2

Abstract

A method by which an electronic device outputs a control signal based on a motion of a user may comprise the steps of: when an operation mode of the electronic device is a gesture recording mode, generating a gesture profile of the user corresponding to a reference motion as the user performs a recording motion of mimicking the reference motion at least once; and when the operation mode is a gesture recognition mode, determining whether a recognition motion of the user matches the reference motion.

Inventors

  • Ho Yeon Kim

Assignees

  • COX SPACE CO., LTD

Dates

Publication Date
20260512
Application Date
20211108
Priority Date
20210912

Claims (13)

  1. 1 . A method for outputting a control signal based on a user's left move motion in an electronic device, the method comprising: acquiring acceleration values for an X-axis in a left-right direction, a Y-axis in an up-down direction, and a Z-axis in a front-rear direction, respectively, based on acceleration data received from an accelerometer of the electronic device; acquiring angular velocity values for a yaw axis, a pitch axis, and a roll axis, respectively, based on gyro data received from a gyroscope of the electronic device; acquiring inclination values for a yaw axis, a pitch axis, and a roll axis, respectively, based on acceleration data, gyro data, and geomagnetic data acquired from the accelerometer, the gyroscope, the magnetometer of the electronic device, respectively; generating, when a current operation mode is a gesture recording mode, the user′ gesture profile corresponding to a reference motion by making a recording motion in which the user makes the reference motion at least once; and determining, when the current operation mode is a gesture recognition mode, whether the user's recognition motion matches the reference motion, wherein the reference motion is a left move, wherein the generating of the user's gesture profile comprises: a first step of determining, when an inclination with respect to the pitch axis of the electronic device is within a first reference range and an inclination with respect to the roll axis is within a second reference range, that the recording motion is ready; a second step of determining, when the recording motion is ready, that the recording motion has started when an angular velocity with respect to the yaw axis is above a first reference value; a third step of ending, when the recording motion is started, the generation of the gesture profile and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of a third reference range or the inclination with respect to the pitch axis is out of a fourth reference range; a fourth step of ending, when an angular velocity with respect to the yaw axis at a completion time of the recording motion is below a second reference value, the generation of the gesture profile and returning to the standby state; a fifth step of determining, when a cumulative value of linear acceleration velocity with respect to the X-axis from a start time to a completion time of the recording motion is below a third reference value, that the recording motion is invalid to end the generation of the gesture profile to return to the standby mode; and a sixth step of storing a reference rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the yaw axis, and a reference interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the yaw axis, in a memory as the user's characteristic values, and wherein the completion time of the recording motion is determined as a time when a current angular velocity value with respect to the yaw axis becomes below a reference ratio of the maximum angular velocity with respect to the yaw axis after the start time of the recording motion.
  2. 2 . The method of claim 1 , wherein the user's gesture profile is generated by the user making the recording motion a number of times, wherein each number is counted only when all operations from the first step to the sixth step are completed, wherein the reference rotation amount is determined based on reference rotation amounts acquired over a number of times, respectively, and wherein the reference interference amount is determined based on reference interference amounts acquired over a number of times, respectively.
  3. 3 . The method of claim 1 , wherein the determining of whether the user's recognition motion matches the reference motion comprises: a seventh step of determining, when an inclination with respect to the pitch axis of the electronic device is within the first reference range and an inclination with respect to the roll axis is within the second reference range, that the recognition motion is ready; an eighth step of determining, when the recognition motion is ready, that the recognition motion has started when an angular velocity with respect to the yaw axis is above the first reference value; a ninth step of ending, when the recognition motion is started, recognition and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of the third reference range or the inclination with respect to the pitch axis is out of the fourth reference range; a tenth step of ending, when an angular velocity with respect to the yaw axis at a completion time of the recognition motion is below the second reference value, recognition and returning to the standby state; an eleventh step of determining, when a cumulative value of linear acceleration velocity with respect to the X-axis from a start time to a completion time of the recognition motion is below the third reference value, that the recognition motion is invalid to end recognition to return to the standby mode; and a twelfth step of comparing a current rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the yaw axis, and a current interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the yaw axis, with the reference rotation amount and the reference interference amount previously stored in the memory, respectively, wherein whether the recognition motion matches the reference motion is determined based on the comparison result, and wherein the completion time of the recognition motion is determined as a time when a current angular velocity value with respect to the yaw axis becomes below a reference ratio of the maximum angular velocity with respect to the yaw axis after the start time of the recognition motion.
  4. 4 . A method for outputting a control signal based on a user's right move motion in an electronic device, the method comprising: acquiring acceleration values for an X-axis in a left-right direction, a Y-axis in an up-down direction, and a Z-axis in a front-rear direction, respectively, based on acceleration data received from an accelerometer of the electronic device; acquiring angular velocity values for a yaw axis, a pitch axis, and a roll axis, respectively, based on gyro data received from a gyroscope of the electronic device; acquiring inclination values for a yaw axis, a pitch axis, and a roll axis, respectively, based on acceleration data, gyro data, and geomagnetic data acquired from the accelerometer, the gyroscope, and the magnetometer of the electronic device, respectively; generating, when a current operation mode is a gesture recording mode, the user′ gesture profile corresponding to a reference motion by making a recording motion in which the user makes the reference motion at least once; and determining, when the current operation mode is a gesture recognition mode, whether the user's recognition motion matches the reference motion, wherein the reference motion is a right move, wherein the generating of the user's gesture profile comprises: a first step of determining, when an inclination value with respect to the roll axis of the electronic device is within a first reference range, that the recording motion is ready; a second step of determining, when the recording motion is ready, that the recording motion has started when an angular velocity with respect to the pitch axis is above a first reference value; a third step of ending, when the recording motion is started, the generation of the gesture profile and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of a second reference range; a fourth step of ending, when an angular velocity with respect to the pitch axis at a completion time of the recording motion is below a second reference value, the generation of the gesture profile and returning to the standby state; a fifth step of determining, when a cumulative value of linear acceleration velocity with respect to the Z-axis from a start time to a completion time of the recording motion is below a third reference value, that the recording motion is invalid to end the generation of the gesture profile to return to the standby mode; and a sixth step of storing a reference rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the pitch axis, and a reference interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the yaw axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the pitch axis, in a memory as the user's characteristic values, and wherein the completion time of the recording motion is determined as a time when a current angular velocity value with respect to the pitch axis becomes below a reference ratio of the maximum angular velocity with respect to the pitch axis after the start time of the recording motion.
  5. 5 . The method of claim 4 , wherein the user's gesture profile is generated by the user making the recording motion a number of times, wherein each number is counted only when all operations from the first step to the sixth step are completed, wherein the reference rotation amount is determined based on reference rotation amounts acquired over a number of times, respectively, and wherein the reference interference amount is determined based on reference interference amounts acquired over a number of times, respectively.
  6. 6 . The method of claim 4 , wherein the determining of whether the user's recognition motion matches the reference motion comprises: a seventh step of determining, when an inclination value with respect to the roll axis of the electronic device is within the first reference range, that the recognition motion is ready; an eighth step of determining, when the recognition motion is ready, that the recognition motion has started when an angular velocity with respect to the pitch axis is above the first reference value; a ninth step of ending, when the recognition motion is started, recognition and returning to a standby state when the inclination with respect to the roll axis of the recognition motion is out of the second reference range; a tenth step of ending, when an angular velocity with respect to the pitch axis at a completion time of the recognition motion is below the second reference value, recognition and returning to the standby state; an eleventh step of determining, when a cumulative value of linear acceleration velocity with respect to the Z-axis from a start time of the recognition motion to a completion time of the recognition motion is below the third reference value, that the recognition motion is invalid to end recognition to return to the standby mode; and a twelfth step of comparing a current rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the pitch axis, and a current interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the yaw axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the pitch axis, with the reference rotation amount and the reference interference amount previously stored in the memory, respectively, wherein whether the recognition motion matches the reference motion is determined based on the comparison result, and wherein the completion time of the recognition motion is determined as a time when a current angular velocity value with respect to the pitch axis becomes below a reference ratio of the maximum angular velocity with respect to the pitch axis after the start time of the recording motion.
  7. 7 . A method for outputting a control signal based on a user's up move motion in an electronic device, the method comprising: acquiring acceleration values for an X-axis in a left-right direction, a Y-axis in an up-down direction, and a Z-axis in a front-rear direction, respectively, based on acceleration data received from an accelerometer of the electronic device; acquiring angular velocity values for a yaw axis, a pitch axis, and a roll axis, respectively, based on gyro data received from a gyroscope of the electronic device; acquiring inclination values for a yaw axis, a pitch axis, and a roll axis, respectively, based on acceleration data, gyro data, and geomagnetic data acquired from the accelerometer, the gyroscope, and the magnetometer of the electronic device, respectively; generating, when a current operation mode is a gesture recording mode, the user′ gesture profile corresponding to a reference motion by making a recording motion in which the user makes the reference motion at least once; and determining, when the current operation mode is a gesture recognition mode, whether the user's recognition motion matches the reference motion, wherein the reference motion is an up move, wherein the generating of the user's gesture profile comprises: a first step of determining, when an inclination with respect to the roll axis of the electronic device is within a first reference range, that the recording motion is ready; a second step of determining, when the recording motion is ready, that the recording motion has started when an angular velocity with respect to the yaw axis is above a first reference value; a third step of ending, when the recording motion is started, the generation of the gesture profile and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of a second reference range; a fourth step of ending, when an angular velocity with respect to the yaw axis at a completion time of the recording motion is below a second reference value, the generation of the gesture profile and returning to the standby state; a fifth step of determining, when a cumulative value of linear acceleration velocity with respect to the X-axis from a start time to a completion time of the recording motion is below a third reference value, that the recording motion is invalid to end the generation of the gesture profile to return to the standby mode; and a sixth step of storing a reference rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the yaw axis, and a reference interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the yaw axis, in a memory as the user's characteristic values, and wherein the completion time of the recording motion is determined as a time when a current angular velocity value with respect to the yaw axis becomes below a reference ratio of the maximum angular velocity with respect to the yaw axis after the start time of the recording motion.
  8. 8 . The method of claim 7 , wherein the user's gesture profile is generated by the user making the recording motion a number of times, wherein each number is counted only when all operations from the first step to the sixth step are completed, wherein the reference rotation amount is determined based on reference rotation amounts acquired over a number of times, respectively, and wherein the reference interference amount is determined based on reference interference amounts acquired over a number of times, respectively.
  9. 9 . The method of claim 7 , wherein the determining of whether the user's recognition motion matches the reference motion comprises: a seventh step of determining, when an inclination with respect to the roll axis of the electronic device is within the first reference range, that the recognition motion is ready; an eighth step of determining, when the recognition motion is ready, that the recognition motion has started when an angular velocity with respect to the yaw axis is above the first reference value; a ninth step of ending, when the recognition motion is started, recognition and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of the second reference range; a tenth step of ending, when an angular velocity with respect to the yaw axis at a completion time of the recognition motion is below the second reference value, recognition and returning to the standby state; an eleventh step of determining, when a cumulative value of linear acceleration velocity with respect to the X-axis from a start time to a completion time of the recognition motion is below the third reference value, that the recognition motion is invalid to end recognition to return to the standby mode; and a twelfth step of comparing a current rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the yaw axis, and a current interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the yaw axis, with the reference rotation amount and the reference interference amount previously stored in the memory, respectively, wherein whether the recognition motion matches the reference motion is determined based on the comparison result, and wherein the completion time of the recognition motion is determined as a time when a current angular velocity value with respect to the yaw axis becomes below a reference ratio of the maximum angular velocity with respect to the yaw axis after the start time of the recognition motion.
  10. 10 . A method for outputting a control signal based on a user's down move motion in an electronic device, the method comprising: acquiring acceleration values for an X-axis in a left-right direction, a Y-axis in an up-down direction, and a Z-axis in a front-rear direction, respectively, based on acceleration data received from an accelerometer of the electronic device; acquiring angular velocity values for a yaw axis, a pitch axis, and a roll axis, respectively, based on gyro data received from a gyroscope of the electronic device; acquiring inclination values for a yaw axis, a pitch axis, and a roll axis, respectively, based on acceleration data, gyro data, and geomagnetic data acquired from the accelerometer, the gyroscope, and the magnetometer of the electronic device, respectively; generating, when a current operation mode is a gesture recording mode, the user′ gesture profile corresponding to a reference motion by making a recording motion in which the user makes the reference motion at least once; and determining, when the current operation mode is a gesture recognition mode, whether the user's recognition motion matches the reference motion, wherein the reference motion is a down move, wherein the generating of the user's gesture profile comprises: a first step of determining, when an inclination with respect to the roll axis of the electronic device is within a first reference range, that the recording motion is ready; a second step of determining, when the recording motion is ready, that the recording motion has started when an angular velocity with respect to the yaw axis is above a first reference value; a third step of ending, when the recording motion is started, the generation of the gesture profile and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of a second reference range or an absolute value of the inclination with respect to the pitch axis is above a second reference value, and the inclination with respect to the roll axis is below a third reference value; a fourth step of ending, when an angular velocity with respect to the yaw axis at a completion time of the recording motion is below a fourth reference value, the generation of the gesture profile and returning to the standby state; a fifth step of determining, when a cumulative value of linear acceleration velocity with respect to the X-axis from a start time to a completion time of the recording motion is below a fifth reference value, that the recording motion is invalid to end the generation of the gesture profile to return to the standby mode; and a sixth step of storing a reference rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the yaw axis, and a reference interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recording motion by a maximum angular velocity with respect to the yaw axis, in a memory as the user's characteristic values, and wherein the completion time of the recording motion is determined as a time when a current angular velocity value with respect to the yaw axis becomes below a reference ratio of the maximum angular velocity with respect to the yaw axis after the start time of the recording motion.
  11. 11 . The method of claim 10 , wherein the user's gesture profile is generated by the user making the recording motion a number of times, wherein each number is counted only when all operations from the first step to the sixth step are completed, wherein the reference rotation amount is determined based on reference rotation amounts acquired over a number of times, respectively, and wherein the reference interference amount is determined based on reference interference amounts acquired over a number of times, respectively.
  12. 12 . The method of claim 10 , wherein the determining of whether the user's recognition motion matches the reference motion comprises: a seventh step of determining, when an inclination with respect to the roll axis of the electronic device is within the first reference range, that the recognition motion is ready; an eighth step of determining, when the recognition motion is ready, that the recognition motion has started when an angular velocity with respect to the yaw axis is above the first reference value; a ninth step of ending, when the recognition motion is started, recognition and returning to a standby state when the inclination with respect to the roll axis of the recording motion is out of the second reference range or an absolute value of the inclination with respect to the pitch axis is above a second reference value, and the inclination with respect to the roll axis is below a third reference value; a tenth step of ending, when an angular velocity with respect to the yaw axis at a completion time of the recognition motion is below the fourth reference value, recognition and returning to the standby state; an eleventh step of determining, when a cumulative value of linear acceleration velocity with respect to the X-axis from a start time to a completion time of the recognition motion is below the fifth reference value, that the recognition motion is invalid to end recognition to return to the standby mode; and a twelfth step of comparing a current rotation amount, which is a value obtained by dividing a maximum angular velocity with respect to the roll axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the yaw axis, and a current interference amount, which is a value obtained by dividing a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recognition motion by a maximum angular velocity with respect to the yaw axis, with the reference rotation amount and the reference interference amount previously stored in the memory, respectively, wherein whether the recognition motion matches the reference motion is determined based on the comparison result, and wherein the completion time of the recognition motion is determined as a time when a current angular velocity value with respect to the yaw axis becomes below a reference ratio of the maximum angular velocity with respect to the yaw axis after the start time of the recognition motion.
  13. 13 . An electronic device for outputting a control signal based on a user's motion, the electronic device comprising: an accelerometer; a gyroscope; a magnetometer; a first processor that acquires acceleration values for an X-axis in a left-right direction, a Y-axis in an up-down direction, and a Z-axis in a front-rear direction, respectively, based on acceleration data received from the accelerometer, acquires angular velocity values for a yaw axis, a pitch axis, and a roll axis, respectively, based on gyro data received from the gyroscope of the electronic device, and acquires inclination values for a yaw axis, a pitch axis, and a roll axis, respectively, based on acceleration data, gyro data, and geomagnetic data acquired from the accelerometer, the gyroscope, and the magnetometer of the electronic device, respectively; a second processor that acquires, at periodic intervals from the first processor, acceleration values for the X-axis in a left-right direction, the Y-axis in an up-down direction, and the Z-axis in a front-rear direction, respectively, angular velocity values for the yaw axis, the pitch axis, and the roll axis, respectively, and inclination values for the yaw axis, the pitch axis, and the roll axis, respectively; and a memory for storing a reference rotation amount and a reference interference amount for the user's recording motion as the user's characteristic values, wherein when a current operation mode is a gesture recording mode, a reference rotation amount and a reference interference amount are acquired from motion data acquired from the recording motion made by the user and stored in the memory, wherein when the current operation mode is a gesture recognition mode, a rotation amount and an interference amount acquired from a recognition motion made by the user are compared with the reference rotation amount and the reference interference amount, respectively, wherein the recording motion comprises a left, right, up, or down move motion, wherein the user's characteristic values comprise a first reference rotation amount and a first reference interference amount for the user's left move motion, a second reference rotation amount and a second reference interference amount for the user's right move motion, a third reference rotation amount and a third reference interference amount for the user's up move motion, and a fourth reference rotation amount and a fourth reference interference amount for the user's down move motion, and wherein when the recording motion is a left, up, or down move motion, the reference rotation amount is a value obtained by dividing a maximum angular velocity with respect to the roll axis by a maximum angular velocity with respect to the yaw axis from a start time to a completion time of the recording motion, and the reference interference amount is a value obtained by dividing a maximum angular velocity with respect to the pitch axis by a maximum angular velocity with respect to the yaw axis from the start time to the completion time of the recording motion, and when the recording motion is a right move motion, the reference rotation amount is a value obtained by dividing a maximum angular velocity with respect to the roll axis by a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recording motion, and the reference interference amount is a value obtained by dividing a maximum angular velocity with respect to the yaw axis by a maximum angular velocity with respect to the pitch axis from the start time to the completion time of the recording motion.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This is a National Stage Application of International Application No. PCT/KR2021/016095 filed Nov. 8, 2021, claiming priority based on Korean Patent Application No. 10-2021-0121363 filed Sep. 12, 2021. TECHNICAL FIELD The present disclosure relates to a gesture recognition method performed in an electronic device. BACKGROUND ART Gesture or motion recognition technology is continuously developing thanks to the growth of wearable devices and the development of the virtual reality-based content market such as computer graphics technology, augmented reality (AR), virtual reality (VR), and mixed reality (MR). However, gesture recognition technology is mainly used to create virtual content in games or content based on computer graphics in movie production companies rather than to control devices, so the gesture recognition technology has very limited use in everyday life. For these reasons, there is a problem in that firstly, it is difficult to use in fields where errors in gesture recognition technology may cause fatal results, and secondly, for gesture recognition, a method of acquiring an image of an object from a camera to recognize a gesture based on image processing is widely used, but the image processing method is greatly influenced by the environment, and thus recognition errors are frequent and the algorithm needs to be very sophisticated. Thirdly, the accuracy of gesture recognition may be increased by having the user attach a landmark or wear a special gear, but this method is too cumbersome and inconvenient for the user, which is why gesture recognition is not used in daily life or daily work. In addition, there are conventional technologies that acquire motion data through a motion sensor and perform gesture recognition based thereon, but most of them are technologies that recognize gestures by using only one or two types of data, such as acceleration data, and processing them. In recent years, a 9-axis sensor including an accelerometer, a gyroscope, and a magnetometer has been widely and inexpensively distributed, but there is no technology to increase the gesture recognition rate by mounting it on a wearable device and recognizing gestures based on the acquired sensor data or generating a gesture profile specialized for a specific user. DISCLOSURE OF INVENTION Technical Problem A method for generating a gesture profile specialized for a user based on motion data acquired by a user's motion on a three-dimensional space, and recognizing the current user's motion as a specific gesture based on the generated gesture profile may be provided. A technical task to be achieved by the present embodiment may not be limited to the above-described technical task, and other technical tasks may be inferred from the following embodiments. Technical Solution A method for generating a gesture profile specialized for a user based on motion data acquired by a user's motion on a three-dimensional space, and recognizing the current user's motion as a specific gesture based on the generated gesture profile may be provided. A technical task to be achieved by the present embodiment may not be limited to the above-described technical task, and other technical tasks may be inferred from the following embodiments. Advantageous Effects A method for generating a gesture profile specialized for a user through acquiring motion data on a three-dimensional space, and recognizing a user's specific gesture based on the generated gesture profile may be provided. As the user directly makes a reference motion, a user characteristic value for a specific reference motion may be stored and gesture recognition is performed based thereon, thereby improving the accuracy of gesture recognition and easily and precisely controlling various content on a smartphone, a TV, a computer, a tablet PC, a hologram, and a head mount display (HMD). BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A shows an electronic device (in a closed state) for controlling a host device, according to one embodiment. FIG. 1B shows an electronic device (in an open state) for controlling a host device, according to one embodiment. FIG. 2A shows a ring-shaped device within an electronic device, according to one embodiment. FIG. 2B shows controlling content using an electronic device, according to one embodiment. FIG. 3 shows a system including an electronic device, according to one embodiment. FIG. 4A shows mouse operations performed with an electronic device under a mouse mode, according to one embodiment. FIG. 4B shows a diagram in which a front portion of the electronic device is divided into three touch regions, according to one embodiment. FIG. 4C shows mouse operations recognized using three regions, according to one embodiment. FIG. 5 shows a decision model for determining a motion corresponding to user's movement information, according to one embodiment. FIG. 6 shows a joystick using an electronic device, according to one embodiment.