Search

KR-20260063663-A - User Interface System and Method Using Hand-Face Hybrid Gesture Recognition

KR20260063663AKR 20260063663 AKR20260063663 AKR 20260063663AKR-20260063663-A

Abstract

The present invention relates to a user interface system using hand-face hybrid gesture recognition, and more specifically, to a user interface system and method for interaction with a user by combining facial and hand gestures. According to the present invention, the invention includes a camera input unit (100) for receiving video data to collect gestures of a user's face and hand in real time, a gesture processing unit (200) for detecting 3D landmarks of the face and hand, determining the presence of the face and hand to set each flag, and detecting the Pan and Tilt angles of the face and the position of the hand to determine the movement of the cursor, a gesture recognition unit (300) for analyzing data detected by the gesture processing unit to recognize face gestures and hand gestures and determining the position of the user's cursor, a control command generation unit (400) for generating user event commands based on the angle and holding time according to the recognizable motion when a face gesture mode or a hand gesture mode is activated based on the gesture data recognized by the gesture recognition unit, and an interface unit (500) for visually displaying the user event commands generated by the control command generation unit and controlling the operation of a target device according to the user event commands, thereby enabling the interface to be controlled freely using the face and hand without additional gestures for switching modes.

Inventors

  • 곽노윤

Assignees

  • 백석대학교산학협력단

Dates

Publication Date
20260507
Application Date
20241030

Claims (8)

  1. A camera input unit (100) for receiving video data to collect user face and hand gestures in real time, A gesture processing unit (200) that detects 3D landmarks of the face and hand, determines the presence of the face and hand to set each flag, and detects the Pan and Tilt angles of the face and the position of the hand to determine the movement of the cursor, A gesture recognition unit (300) that analyzes data detected by the gesture processing unit to recognize face gestures and hand gestures and determines the user's cursor position, A control command generating unit (400) that generates a user event command according to an angle and holding time based on a recognizable motion when a face gesture mode or a hand gesture mode is activated based on gesture data recognized by the gesture recognition unit above, and It includes an interface unit (500) that visually displays a user event command generated by the control command generation unit and controls the operation of a target device according to the user event command, and A user interface system using hand-face hybrid gesture recognition, characterized in that the control command generation unit generates a user event command based on a hand gesture after switching to a hand gesture mode due to the failure to recognize the angle and holding time according to the operation in face gesture mode.
  2. In paragraph 1, The gesture processing unit (200) above is, A 3D landmark detection module (210) that detects 3D landmarks of the face and hand in image data collected from the camera input unit, A flag setting module (220) that analyzes detected 3D landmark data to determine whether a face and a hand exist within the image data, sets a face presence flag to active (True) if a face is detected, and sets a hand presence flag to active (True) if a hand is detected. A user interface system using hand-face hybrid gesture recognition, characterized by including a cursor information module (230) that detects the pan and tilt angles of the face and the position of the hand to determine the left-right and up-down movement of the cursor.
  3. In paragraph 1, The gesture recognition unit (300) above is, A data analysis module (310) that receives 3D landmark data of the face and hand detected by the gesture processing unit and analyzes the Pan and Tilt angles of the face and the position data of the hand, A recognition module (320) that analyzes face gestures and hand gestures to set a face gesture mode or a hand gesture mode, and when the face and hand are simultaneously present within the input frame, mixes the two gestures to set the gesture so that it can be recognized regardless of the mode. A user interface system using hand-face hybrid gesture recognition, characterized by including a cursor positioning module (330) that determines the left-right and up-down movement of the cursor based on the recognized gesture and the pan and tilt angles of the face, and performs fine adjustment of the cursor using the position of the hand.
  4. In paragraph 1, The above control command generating unit (400) is, An interface activation module (410) that determines whether gesture recognition is possible by switching the hybrid gesture interface activation flag to an activated state through the gesture processing unit based on the presence of the face and hand, When the above hybrid gesture interface activation flag is in an active state, an event command generation module (420) generates a user event command according to the angle and holding time of a recognizable motion and When the above user event command is 'gesture mode switching', it includes a user event processing module (430) that changes the gesture mode and generates a command to assign a cursor position corresponding to the gesture and process a user event according to the gesture. A user interface system using hand-face hybrid gesture recognition, characterized by further including an interface deactivation module (440) that switches the hybrid gesture interface activation flag to deactivation when an interface deactivation event is input while processing the user event in the hybrid gesture interface activation state.
  5. In a method using a user interface system utilizing hand-face hybrid gesture recognition, (a) A step in which the above user interface system receives video data to collect user face and hand gestures in real time through a camera input unit, (b) A step in which the user interface system detects 3D landmarks of the face and hand through a gesture processing unit, determines the presence of the face and hand to set each flag, and detects the Pan and Tilt angles of the face and the position of the hand to determine the movement of the cursor, (c) A step in which the user interface system analyzes data detected by the gesture processing unit through a gesture recognition unit to recognize face gestures and hand gestures and determines the position of the user's cursor, (d) When the user interface system is activated in face gesture mode or hand gesture mode through a control command generation unit based on gesture data recognized by the gesture recognition unit, a step of generating a user event command according to an angle and holding time according to a recognizable movement and (e) The above user interface system includes the step of visually displaying a user event command through an interface unit and controlling the operation of a target device according to the user event command, The above step (d) is a method using a user interface system utilizing hand-face hybrid gesture recognition, characterized by switching to hand gesture mode due to non-recognition of the angle and holding time according to the movement in face gesture mode, and generating a user event command based on the hand gesture.
  6. In paragraph 5, The above step (b) is, The gesture processing unit detects 3D landmarks of the face and hand in image data collected from the camera input unit, The gesture processing unit analyzes the detected 3D landmark data to determine whether a face and a hand exist within the image data, and if a face is detected, sets the face presence flag to activated (True), and if a hand is detected, sets the hand presence flag to activated (True). A method using a user interface system utilizing hand-face hybrid gesture recognition, characterized in that the gesture processing unit includes the step of detecting the pan and tilt angles of the face and the position of the hand to determine the left-right and up-down movement of the cursor.
  7. In paragraph 5, The above step (c) is, A step in which the gesture recognition unit receives 3D landmark data of the face and hand detected by the gesture processing unit and analyzes the pan and tilt angles of the face and the position data of the hand, The gesture recognition unit analyzes a face gesture and a hand gesture to set a face gesture mode or a hand gesture mode, and when a face and a hand are simultaneously present within an input frame, the two gestures are mixed to enable gesture recognition regardless of the mode. A method using a user interface system utilizing hand-face hybrid gesture recognition, characterized in that the gesture recognition unit determines the left-right and up-down movement of the cursor based on the recognized gesture and the pan and tilt angles of the face, and performs fine adjustment of the cursor using the position of the hand.
  8. In paragraph 5, The above step (d) is, A step in which the control command generating unit determines whether gesture recognition is possible by switching the hybrid gesture interface activation flag to an activated state through the gesture processing unit based on the flag setting according to the presence of a face and a hand, The above control command generation unit generates a user event command according to an angle and holding time based on a recognizable motion when the hybrid gesture interface activation flag is in an activated state, and The above control command generating unit includes the step of generating a command that processes a cursor position corresponding to the gesture and a user event according to the gesture by changing the gesture mode when the user event command is 'gesture mode switching'. A method using a user interface system utilizing hand-face hybrid gesture recognition, characterized by further including the step of switching a hybrid gesture interface activation flag to deactivation when an interface deactivation event is input while processing a user event in the hybrid gesture interface activation state.

Description

User Interface System and Method Using Hand-Face Hybrid Gesture Recognition The present invention relates to a user interface system using hand-face hybrid gesture recognition, and more specifically, to a user interface system and method for interaction with a user by combining facial and hand gestures. Human-Computer Interaction (HCI) technology has been continuously innovating to enhance natural communication between humans and computers, thanks to rapid advancements in sensors, artificial intelligence, CPUs, GPUs, and memory. In particular, new Natural User Interface (NUI) technologies are being actively researched to control digital devices by utilizing biosignals—such as electromyography (EMG), electrocardiography (ECG), brainwaves, pulse waves, and electrooculography—in addition to voice, gaze, facial expressions, gestures, and touch. NUI technology aims to break down physical barriers between humans and machines and provide an intuitive and natural user experience. It enables devices to operate appropriately and process tasks efficiently by reflecting the user's cognitive or physical state and situation. Human gesture recognition technology is a core research area of NUI, and since the 1990s, its scope has been broad and expanding day by day, encompassing applications such as human-robot interaction, 3D game interfaces, virtual reality, consumer electronics interaction, medical posture correction and momentum measurement, sign language recognition, drone control, MANO, Fast Hand, DIGIT, floating holographic character control, and MVHM in the field of immersive media. As Google’s MediaPipe emerges as the center of cross-platform frameworks in the field of gesture recognition, it helps to implement gesture recognition technology more easily, and hand gesture interface technology based on the MediaPipe Hands model is being utilized in various application fields. Recently, user interfaces have been evolving in a direction that is more intuitive and user-friendly. In particular, active research is being conducted on technologies that enable users to control devices more naturally through gesture recognition. While existing gesture recognition technologies have primarily focused on specific body parts such as the hands or face, recognizing a single gesture has limitations in executing complex commands. Consequently, there is a growing need for hybrid gesture interfaces that combine facial and hand gestures to enable more accurate and diverse control. FIG. 1 is a diagram showing a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating the operation of a user interface system using hand-face hybrid gesture recognition according to an embodiment of the present invention. FIG. 3 shows a 3D hand landmark of a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. FIG. 4 shows a 3D face landmark of a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. Figure 5 shows the Pan angle and Tilt angle of a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. Figure 6 shows the x-axis pixel position values on the screen for the face pan angle of a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. Figure 7 shows the Roll angle of a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. Figure 8 is a diagram illustrating the determination of hand bending using two vectors. FIG. 9 shows the hand shape used in the proposed hand gesture recognition in a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. FIG. 10 shows a dial gesture that generates a volume control event of a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. FIG. 11 shows a user scenario simulation using a hand-face hybrid gesture interface proposed in one embodiment of the present invention. According to one embodiment of the present invention, the system is a user interface system using hand-face hybrid gesture recognition that recognizes a user's face and hand gestures in real time using a MediaPipe Face Mesh model and a MediaPipe Hands model, and generates a cursor position and user commands based thereon to implement a hand-face hybrid gesture-based user interface. Hereinafter, a user interface system using hand-face hybrid gesture recognition according to an embodiment of the present invention will be described with reference to the attached drawings. FIG. 1 is a diagram showing a user interface system using hand-face hybrid gesture recognition according to one embodiment of the present invention. A