Search

CN-122018704-A - Dual-mode human-computer interaction method, system, equipment and medium based on touch micro-fibrillation modulation

CN122018704ACN 122018704 ACN122018704 ACN 122018704ACN-122018704-A

Abstract

The invention discloses a bimodal man-machine interaction method, a system, equipment and a medium based on touch micro-fibrillation modulation, belonging to the technical field of man-machine interaction, wherein the method comprises the steps of obtaining a reference micro-fibrillation signal of a user, extracting a reference characteristic of the user, collecting an actual measurement touch signal of the user during normal interactive operation, identifying a touch instruction of the user, and extracting actual measurement micro-fibrillation characteristics, including actual measurement basic parameters and micro-fibrillation modes; and identifying a plurality of candidate micro-fibrillation modulation instructions according to the mapping relation between the micro-fibrillation mode and the candidate micro-fibrillation modulation instructions, and between the actually measured basic parameters and the user reference characteristic deviation amounts and the candidate micro-fibrillation modulation instructions, and performing multiple screening to obtain an optimal micro-fibrillation modulation instruction, and fusing the touch control instruction of the user and the optimal micro-fibrillation modulation instruction to obtain a final interaction instruction. The invention can introduce conscious micro-fibrillation characteristics of touch control into human-computer interaction, effectively improves human-computer interaction efficiency, and is suitable for common users and hand dyskinesia users.

Inventors

  • WANG KE

Assignees

  • 江苏开放大学（江苏城市职业学院）

Dates

Publication Date
20260512
Application Date
20260415

Claims (10)

  1. 1. A bimodal man-machine interaction method based on touch micro-fibrillation modulation is characterized by comprising the following steps: Acquiring a reference micro-fibrillation signal when a user performs free sliding operation on a touch interface, and extracting user reference characteristics of the reference micro-fibrillation signal; Collecting actual measurement touch signals of a user when the user performs normal interactive operation on a touch interface, and identifying a touch instruction of the user; After a micro-fibrillation modulating signal is separated from an actually-measured touch signal, the actually-measured micro-fibrillation characteristic of the micro-fibrillation modulating signal is extracted, wherein the actually-measured micro-fibrillation characteristic comprises actually-measured basic parameters and a micro-fibrillation mode; Respectively constructing a mapping relation between a micro-fibrillation mode and a candidate micro-fibrillation modulation instruction and a mapping relation between an actually measured basic parameter and a user reference characteristic deviation value and the candidate micro-fibrillation modulation instruction, obtaining a candidate micro-fibrillation modulation instruction corresponding to the current micro-fibrillation mode and a candidate micro-fibrillation modulation instruction corresponding to the current actually measured basic parameter and the user reference characteristic deviation value, and obtaining an optimal micro-fibrillation modulation instruction by multiple screening of all the candidate micro-fibrillation modulation instructions; Based on a preset fusion rule, fusing the touch control instruction of the user and the optimal micro-fibrillation modulation instruction to obtain a final interaction instruction.
  2. 2. The method for bimodal human-computer interaction based on touch micro-fibrillation modulation according to claim 1, wherein the user reference features comprise a reference micro-fibrillation frequency and a reference micro-fibrillation amplitude, and the actual measurement basic parameters comprise an actual measurement micro-fibrillation main frequency and an actual measurement micro-fibrillation amplitude.
  3. 3. The method for dual-mode human-computer interaction based on touch-control micro-fibrillation modulation according to claim 2, wherein the extracting the actually measured micro-fibrillation characteristic of the micro-fibrillation modulation signal specifically comprises: And (3) carrying out time-frequency analysis on the micro-fibrillation modulation signal by utilizing a sliding window and adopting an FFT method to obtain the actual measurement micro-fibrillation main frequency of the micro-fibrillation modulation signal, wherein the specific formula is as follows: ; Wherein, the For the moment of time The measured main frequency of the micro-fibrillation modulated signal, As a function of the frequency variation, In order to perform the fast fourier transform operation, Representing the square of the modulus, For the moment of time Is provided with a micro-fibrillation modulation signal of amplitude, As a function of the integration time variable, Is the sliding window length; calculating the actual measured micro-fibrillation amplitude of the micro-fibrillation modulation signal: Wherein, the method comprises the steps of, For the moment of time The measured micropibrillation amplitude of the micropibrillation modulation signal; constructing a micro-fibrillation activation threshold according to the user reference characteristics by adopting the following steps ; ; ; Wherein, the For a reference micro-fibrillation amplitude of the user, Is an index of the stability of the micro-fibrillation frequency, Is the standard deviation of the reference micro-fibrillation signal frequency sequence, A reference micro-fibrillation frequency for the user; based on a micro-fibrillation activation threshold Defining a micro-fibrillation state of the micro-fibrillation modulation signal according to the following formula; ; Wherein, the For the moment of time A micro-fibrillation state of 0 or 1,1 representing active, 0 representing silence; dividing a micro-fibrillation modulated signal within a sliding window into Each micro-fibrillation segment has the same micro-fibrillation state to construct the micro-fibrillation mode of the micro-fibrillation modulation signal : Wherein, the method comprises the steps of, Is the first The micro-fibrillation state of the individual micro-fibrillation fragments, Is the first Duration of each micro-fibrillation fragment.
  4. 4. The method for dual-mode man-machine interaction based on touch micro-fibrillation modulation according to claim 2, wherein the obtaining the candidate micro-fibrillation modulation command corresponding to the current micro-fibrillation mode and the candidate micro-fibrillation modulation command corresponding to the current measured basic parameter and the user reference characteristic deviation amount specifically comprises: calculating the frequency offset of the actual micro-fibrillation main frequency relative to the reference micro-fibrillation frequency Based on frequency offset Mapping relation with candidate micro-fibrillation modulation instruction to obtain frequency offset The corresponding first candidate micro-fibrillation modulation instruction; Calculating the amplitude change rate of the actual micro-fibrillation amplitude relative to the reference micro-fibrillation amplitude Based on the rate of change of amplitude Mapping relation with candidate micro-fibrillation modulation instruction to obtain amplitude change rate A corresponding second candidate micro-fibrillation modulation instruction; and acquiring a third candidate micro-fibrillation modulation instruction corresponding to the micro-fibrillation mode according to the micro-fibrillation mode of the micro-fibrillation modulation signal and the mapping relation between the micro-fibrillation mode and the candidate micro-fibrillation modulation instruction.
  5. 5. The method for dual-mode human-computer interaction based on touch micro-fibrillation modulation according to claim 4, wherein the method is characterized in that the optimal micro-fibrillation modulation instruction is obtained by performing multiple screening on all candidate micro-fibrillation modulation instructions, and specifically comprises the following steps: Performing quadruple validity verification on all candidate micro-fibrillation modulation instructions, wherein the quadruple validity verification comprises feature duration verification, feature variation amplitude verification, confidence threshold verification and user intention verification; When the confidence threshold value verification is carried out, the comprehensive confidence coefficient of each candidate micro-fibrillation modulation instruction is calculated : Wherein, the method comprises the steps of, As a result of the confidence weighting coefficients, For the feature matching degree of the candidate micro-fibrillation modulation instruction, Obtaining the correlation between the current application scene and the current candidate micro-fibrillation modulation instruction for context consistency; Calculating the feature matching degree of the first candidate micro-fibrillation modulation instruction according to the following formula ; Wherein, the method comprises the steps of, A threshold value for a set frequency offset; Calculating the feature matching degree of the second candidate micro-fibrillation modulation instruction according to the following formula : Wherein, the method comprises the steps of, A threshold value for a set amplitude change rate; feature matching degree of third candidate micro-fibrillation modulation instruction Acquiring through assignment; Counting the number of candidate micro-fibrillation modulation instructions passing through quadruple validity verification, namely outputting no instruction as an optimal micro-fibrillation modulation instruction if the number of candidate micro-fibrillation modulation instructions is 0, taking the candidate micro-fibrillation modulation instructions as the optimal micro-fibrillation modulation instruction if the number of candidate micro-fibrillation modulation instructions is only 1, and carrying out three rounds of priority screening if the number of candidate micro-fibrillation modulation instructions is multiple, wherein the first round of priority screening is that the characteristic matching degree of the candidate micro-fibrillation modulation instructions is highest, the second round of priority screening is that the context consistency of the candidate micro-fibrillation modulation instructions is highest, and the third round of priority screening is that the historical operation association of the candidate micro-fibrillation modulation instructions is highest; The three rounds of priority screening is carried out, specifically, screening is carried out round by round according to preset priority, after each round is finished, if only 1 or 0 candidate micro-fibrillation modulation instructions are left, the corresponding optimal micro-fibrillation modulation instructions are output, otherwise, the next round of screening is carried out, after the three rounds of screening are finished, if only 1 candidate micro-fibrillation modulation instructions are left, the candidate micro-fibrillation modulation instructions are output, otherwise, no instructions are output as the optimal micro-fibrillation modulation instructions.
  6. 6. The method for bimodal man-machine interaction based on touch micro-fibrillation modulation according to claim 1, further comprising separating a reference micro-fibrillation signal from a reference touch signal when a user performs free sliding operation on a touch interface, and performing the following processing on the reference touch signal or an actually measured touch signal when the reference micro-fibrillation signal or the micro-fibrillation modulation signal is separated: Calculating a displacement sequence of the reference touch signal or the actually measured touch signal according to the following steps, and performing high-pass filtering on the displacement sequence to obtain a reference micro-fibrillation signal or a micro-fibrillation modulation signal: ; ; ; Wherein, the 、 The first touch signal is a reference touch signal or an actual measurement touch signal respectively At a plurality of sampling points Direction and direction A directional displacement component; is the first The displacement amplitude of the individual sampling points.
  7. 7. The method of claim 1, further comprising determining whether a time period from the reference micro-fibrillation signal acquisition time to the present time exceeds a set period, if so, automatically updating the reference micro-fibrillation signal, and performing weighted summation on a user reference feature corresponding to the updated reference micro-fibrillation signal and a user reference feature before updating to obtain an updated user reference feature, otherwise, maintaining the current user reference feature.
  8. 8. A bimodal man-machine interaction system based on touch-control micro-fibrillation modulation is characterized by comprising: the reference characteristic acquisition module is used for acquiring a reference micro-fibrillation signal when a user performs free sliding operation on the touch interface and extracting user reference characteristics of the reference micro-fibrillation signal; The touch instruction identification module is used for acquiring actual measurement touch signals of a user when the user performs normal interactive operation on the touch interface and identifying touch instructions of the user; the actual measurement characteristic acquisition module is used for extracting actual measurement micro-fibrillation characteristics of the micro-fibrillation modulation signals after the micro-fibrillation modulation signals are separated from the actual measurement touch control signals, and the actual measurement characteristics comprise actual measurement basic parameters and micro-fibrillation modes; The micro-fibrillation modulation instruction acquisition module is used for respectively constructing a mapping relation between a micro-fibrillation mode and a candidate micro-fibrillation modulation instruction and a mapping relation between an actually measured basic parameter and a user reference characteristic deviation amount and the candidate micro-fibrillation modulation instruction, obtaining the candidate micro-fibrillation modulation instruction corresponding to the current micro-fibrillation mode and the candidate micro-fibrillation modulation instruction corresponding to the current actually measured basic parameter and the user reference characteristic deviation amount, and obtaining the optimal micro-fibrillation modulation instruction by carrying out multiple screening on all the candidate micro-fibrillation modulation instructions; And the double-instruction fusion module is used for fusing the touch instruction of the user and the optimal micro-fibrillation modulation instruction based on a preset fusion rule to obtain a final interaction instruction.
  9. 9. A computer device comprising a processor and a memory, wherein the processor, when executing a computer program stored in the memory, implements the steps of the touch-based micro-fibrillation-modulated bimodal human-machine interaction method as claimed in any one of claims 1-7.
  10. 10. A computer readable storage medium for storing a computer program which when executed by a processor implements the steps of the touch-based micro-fibrillation modulation bimodal human-machine interaction method as claimed in any one of claims 1-7.

Description

Dual-mode human-computer interaction method, system, equipment and medium based on touch micro-fibrillation modulation Technical Field The invention belongs to the technical field of human-computer interaction, and particularly relates to a touch-control micro-fibrillation modulation-based bimodal human-computer interaction method, a system, equipment and a medium. Background Touch screens have become a primary input mode of modern intelligent terminals, and are widely applied to smart phones, tablet computers, smart watches, industrial control panels and other devices. During the touch operation, the user's finger may generate a micro-tremor phenomenon, which is called physiological tremor (hereinafter referred to as micro-tremor) in medicine, and its frequency is generally distributed in the range of 4-12Hz, and its amplitude is generally between 0.1-0.5 mm. Physiological tremor is an intrinsic property of the human neuromuscular system, produced by the rhythmic activity of the central nervous system in combination with peripheral reflex mechanisms. Studies have shown that even in a resting state, there is a weak rhythmic vibration in the fingers of healthy adults. When performing fine touch operations, this micro-vibration is superimposed on the intended motion, forming a high frequency oscillating component in the touch trajectory. The prior art has high consistency in the processing thought of the touch micro-fibrillation, namely the micro-fibrillation is regarded as noise or interference to be eliminated. The specific technical route comprises (1) filtering elimination technology, namely filtering high-frequency micro-fibrillation components by low-pass filtering, self-adaptive filtering and other methods, and reserving low-frequency intended motion tracks. The method is simple and direct, but can lose part of the effective information. (2) And the machine learning denoising technology is to learn the micro-fibrillation characteristics by using methods such as deep learning and the like so as to realize intelligent denoising. This method requires a large amount of training data and has limited generalization ability of the model. (3) And the prediction supplementing technology is used for estimating and compensating touch control deviation caused by micro-fibrillation based on a prediction model of the user intended motion. The method needs to establish an accurate motion model, and has high calculation complexity. However, the prior art has the following disadvantages: The prior touch interaction system essentially eliminates the micro-fibrillation, ignores the application value of the micro-fibrillation, has certain control capability for the micro-fibrillation characteristics of most users (including users with pathological tremors), can consciously adjust the frequency, amplitude and mode of the tremors, and wastes the micro-fibrillation resources by adopting an elimination mode for the micro-fibrillation. In addition, human-computer interaction is performed using only a single dimension of the finger movement position, resulting in relatively low human-computer interaction efficiency, especially for hand dyskinesia users. Disclosure of Invention Aiming at the defects in the prior art, the invention provides a bimodal man-machine interaction method, a bimodal man-machine interaction system, bimodal man-machine interaction equipment and a bimodal man-machine interaction medium based on touch micro-fibrillation modulation, which can introduce conscious micro-fibrillation characteristics of touch into man-machine interaction, effectively improve man-machine interaction efficiency and are suitable for common users and hand dyskinesia users. The invention provides the following technical scheme: in a first aspect, a dual-mode human-computer interaction method based on touch micro-fibrillation modulation is provided, including: Acquiring a reference micro-fibrillation signal when a user performs free sliding operation on a touch interface, and extracting user reference characteristics of the reference micro-fibrillation signal; Collecting actual measurement touch signals of a user when the user performs normal interactive operation on a touch interface, and identifying a touch instruction of the user; After a micro-fibrillation modulating signal is separated from an actually-measured touch signal, the actually-measured micro-fibrillation characteristic of the micro-fibrillation modulating signal is extracted, wherein the actually-measured micro-fibrillation characteristic comprises actually-measured basic parameters and a micro-fibrillation mode; Respectively constructing a mapping relation between a micro-fibrillation mode and a candidate micro-fibrillation modulation instruction and a mapping relation between an actually measured basic parameter and a user reference characteristic deviation value and the candidate micro-fibrillation modulation instruction, obtaining a candidate micro-fibrillation modu