CN-122018698-A - VR/AR interaction method based on multi-granularity micro-gestures

CN122018698ACN 122018698 ACN122018698 ACN 122018698ACN-122018698-A

Abstract

According to the VR/AR interaction method based on the multi-granularity micro-gestures, the traditional thumb-index finger micro-gestures are expanded to thumb-multi-finger combinations, and gesture granularity levels are introduced, so that the expression capability of the micro-gestures in AR/VR complex interaction tasks is effectively improved. The gesture granularity structure enables the micro gesture space to be increased from a limited space into a hierarchical and extensible action system, so that the dexterity of multi-finger collaborative input is enhanced, the limitation that the traditional method cannot cover complex tasks due to limited gesture quantity is avoided, and a richer and more efficient input mode is provided.

Inventors

REN PENGFEI
WANG JINGYU
SUN HAIFENG
PAN TING
LIU XINGYU
WANG JING

Assignees

北京邮电大学

Dates

Publication Date: 20260512
Application Date: 20260224

Claims (1)

1. The VR/AR interaction method based on the multi-granularity micro-gestures is characterized by comprising the following operation steps: (1) The traditional single combination of the thumb and the index finger is expanded into multi-finger and flexible combination of the thumb, the middle finger, the ring finger and the little finger, unified, easy-to-operate and high-efficiency micro-gesture actions are introduced on the basis, wherein the micro-gesture actions comprise finger tip single click, finger tip double click and thumb sliding up and down and left and right along the side edges of other fingers; (2) The system guides the user to execute all micro-gestures at a natural speed and collect hand skeleton sequence data before the user formally starts interaction, then calculates gesture characteristics of the user through a graph convolution network, constructs gesture distribution codes of the user, and injects the codes serving as user characteristic labels into a micro-gesture recognition network so as to realize personalized micro-gesture recognition; (3) Abstracting and classifying user operations from the view of interactive intention to construct a task level model, wherein the task level model extracts intention attributes in the interactive operation, namely, taking an atomic operation which is not subdivided such as click sliding and the like as a basic task, taking a target guiding operation which is switched to other pages or is used for searching specific information and the like as a navigation task, taking system configuration operations such as ambient light or screen size and the like as a system setting task, taking global perception operations such as a view angle or interactive layout of a user and the like as a global view task, and inducing complex AR/VR operations into intention tasks with clear structure and clear boundaries through the abstraction of the interactive intention to construct a highly generalized task level model; (4) On the basis of the task level model, further analyzing the execution frequencies of different task types to find that the task level model has stable and consistent distribution characteristics, namely, the basic tasks are most frequently used, and then navigation tasks, system setting tasks and global view tasks are sequentially performed, so that obvious decreasing trend is presented, and a task level structure arranged according to the execution frequencies is constructed based on the characteristics; (5) And performing collaborative mapping on the task hierarchical structure and the micro gesture granularity hierarchical structure to enable the high-granularity micro gesture to correspond to a high-frequency task and the low-granularity micro gesture to correspond to a low-frequency task, so that an interaction system which accords with human engineering and user cognition rules is formed, and a VR/AR interaction mode with high precision, low delay and low fatigue is realized by using the interaction system.

Description

VR/AR interaction method based on multi-granularity micro-gestures Technical Field The invention relates to a VR/AR interaction method based on multi-granularity micro gestures, belongs to the technical field of information, and particularly belongs to the technical field of VR/AR interaction. Background With the rapid development of Virtual Reality (VR) and augmented Reality (Augmented Reality, AR), a Three-dimensional user interface (Three-Dimensional User Interface,3D UI) is not only used for carrying and presenting multi-source information, but also a core interaction carrier for a user to perform operations, manage applications and complete tasks. With the continuous expansion of the number of applications, interface levels and window layouts, users need to frequently switch among multiple 3D UIs to browse contents, adjust system states or control virtual objects, and such complex interaction scenarios have higher requirements on the efficiency, stability, light weight and usability of the interaction mode. The main input modes in the current AR/VR system include key interaction based on a handle Controller (Controller), bare hand interaction based on visual hand tracking, and micro-gesture (Microgestures) interaction based on finger micro-actions developed in recent years, which together form the main stream interaction mode of the current AR/VR. The main stream interactive mode is characterized in that the handle controller is interactively dependent on entity keys and high-precision tracking, so that stable and continuous input can be realized. The bare hand interaction avoids equipment dependence through a visual hand tracking technology, and enables interaction to be more natural and smooth, and mainly comprises three types, namely, long-range ray interaction, namely, virtual rays are sent out from the palm or finger direction, selection and confirmation are completed through cooperation of thumb and index finger, short-range touch interaction, namely, the hand of a user is instantiated as a virtual hand, clicking, sliding and dragging similar to a touch screen are achieved, micro gesture interaction is carried out, fine movements of fingers are used as input, and the touch screen is small in movement amplitude, strong in concealment and suitable for long-time use. However, in increasingly complex interaction scenarios, high-frequency switching and high-precision operation are required, and the above methods have the limitations that the functional key mapping of the handle controller aggravates the memory burden of a user, the portability and flexibility are limited by holding the physical device, the bare hand rays and touch interaction require larger arm movements, the bare hand rays and touch interaction are easy to cause fatigue after long-time use and have insufficient stability in fine tasks, most of the existing micro gestures adopt single combination of thumb and index finger for interaction, the gesture space is limited, and the increasingly complex interaction tasks are difficult to cover. How to overcome the defects of the current mainstream interaction mode, a high-precision, low-delay and low-fatigue micro-gesture interaction system is designed to more effectively meet the requirements of multi-interface and multi-task parallel operation in an AR/VR environment, and the method becomes a technical problem to be solved in the technical field of VR/AR interaction. Disclosure of Invention In view of the above, the present invention aims to invent a method for systematically introducing thumb and multi-finger combinations into micro-gesture interaction in an AR/VR scene, so as to break through the problem of limited micro-gesture space caused by the dependence of thumb-index finger combinations in the prior art. In order to achieve the above purpose, the invention provides a VR/AR interaction method based on multi-granularity micro-gestures, which comprises the following operation steps: (1) The traditional single combination of the thumb and the index finger is expanded into multi-finger and flexible combination of the thumb, the middle finger, the ring finger and the little finger, unified, easy-to-operate and high-efficiency micro-gesture actions are introduced on the basis, wherein the micro-gesture actions comprise finger tip single click, finger tip double click and thumb sliding up and down and left and right along the side edges of other fingers; (2) The system guides the user to execute all micro-gestures at a natural speed and collect hand skeleton sequence data before the user formally starts interaction, then calculates gesture characteristics of the user through a graph convolution network, constructs gesture distribution codes of the user, and injects the codes serving as user characteristic labels into a micro-gesture recognition network so as to realize personalized micro-gesture recognition; (3) Abstracting and classifying user operations from the view of interactive inte