CN-121982069-A - Visual image-based upper limb movement intention recognition and track prediction method and system

CN121982069ACN 121982069 ACN121982069 ACN 121982069ACN-121982069-A

Abstract

The invention discloses a visual image-based upper limb movement intention recognition and track prediction method and a visual image-based upper limb movement intention recognition and track prediction system, which belong to the technical field of visual and pattern recognition. According to the invention, the two-dimensional pixel coordinates and the pixel depth information of the key points of the upper limbs are fused to construct the three-dimensional space position coordinates, and the time sequence characteristics of the upper limbs are extracted on the basis, so that the accumulated influence of the extraction errors of the key points on the time dimension is effectively inhibited. Further, by introducing a time sequence alignment and motion characteristic matching mechanism, accurate judgment of the upper limb motion intention is realized, and the upper limb motion trail of the next period is predicted according to the accurate judgment. Meanwhile, the predicted track is verified and adjusted by combining the preset upper limb movement constraint conditions, so that the stability of upper limb movement intention recognition and the accuracy of track prediction in the rehabilitation training process are effectively improved.

Inventors

WANG JINGLUAN
WANG XUPENG
LI ZE

Assignees

西安理工大学

Dates

Publication Date: 20260505
Application Date: 20260408

Claims (10)

1. The upper limb movement intention recognition and track prediction method based on the visual image is characterized by comprising the following steps of: acquiring a visual image sequence of the upper limb movement process of a target object in a current period, and extracting two-dimensional pixel coordinates of key points of the upper limb; Acquiring a pixel depth value of an upper limb key point, fusing the pixel depth value with a two-dimensional pixel coordinate to obtain a three-dimensional space position coordinate, extracting time sequence features of upper limb movement based on the three-dimensional space position coordinate, and obtaining time sequence feature data for representing the change features of the movement direction and the movement trend features of the upper limb of the target object; performing time sequence alignment and motion feature matching based on dynamic time warping on time sequence feature data of the current time period and time sequence feature data of the previous time period, and calculating the matching confidence between the upper limb motion intention determined in the previous time period and the upper limb motion track change in the current time period; Determining the upper limb movement intention of the target object in the current period based on the matching confidence degree, the time sequence characteristic data of the current period and a preset intention judging rule, and predicting the upper limb movement locus of the target object in the next period according to the upper limb movement intention of the current period to obtain an initial prediction locus; Judging whether the initial predicted track meets the preset upper limb movement constraint condition, if so, outputting the initial predicted track as a target predicted track, and if not, outputting the target predicted track after adjusting the initial predicted track.
2. The visual image-based upper limb movement intention recognition and trajectory prediction method according to claim 1, wherein the extracting two-dimensional pixel coordinates of the upper limb key points further comprises: Respectively constructing a corresponding time sequence coordinate sequence based on two-dimensional pixel coordinates of each upper limb key point, wherein the upper limb key points at least comprise shoulder key points, elbow key points and wrist key points; Performing abnormal jump detection on the time sequence coordinate sequences of the upper limb key points, calculating coordinate displacement of the same upper limb key point between adjacent frames in the current period, and judging that the two-dimensional pixel coordinates of the upper limb key point corresponding to the current frame are abnormal data when the coordinate displacement is larger than a preset abrupt change threshold; And carrying out interpolation processing on the abnormal data by adopting a linear interpolation algorithm to correct the time sequence coordinate sequence, carrying out smoothing processing on the corrected time sequence coordinate sequence by adopting a Kalman filtering algorithm, and updating the two-dimensional pixel coordinates of each upper limb key point.
3. The visual image-based upper limb movement intention recognition and track prediction method according to claim 2, wherein the extracting the time sequence feature of the upper limb movement based on the three-dimensional space position coordinates comprises the following specific steps: Constructing a local coordinate system by taking shoulder key points of a target object as an origin, and converting three-dimensional space position coordinates of key points of each upper limb into the local coordinate system to obtain relative three-dimensional space position information for eliminating trunk motion interference; constructing an upper limb skeleton vector based on the relative three-dimensional space position information, wherein the upper limb skeleton vector comprises an upper arm vector from a shoulder key point to an elbow key point and a forearm vector from the elbow key point to a wrist key point; Calculating the space velocity of each upper limb skeleton vector in a preset time window to obtain a space velocity characteristic sequence, and calculating the angular velocity of an included angle between the upper limb skeleton vectors to obtain an angular velocity sequence; And fusing the space velocity characteristic sequence and the angular velocity sequence to obtain time sequence characteristic data.
4. The visual image-based upper limb movement intention recognition and trajectory prediction method according to claim 3, wherein the step of performing dynamic time warping-based time series alignment and movement feature matching on the time series feature data of the current period and the previous period specifically comprises the steps of: Constructing a Euclidean distance matrix between time sequence feature data of a current time period and time sequence feature data of a previous time period, and searching an optimal bending path in the Euclidean distance matrix by utilizing a dynamic time warping algorithm, wherein the optimal bending path is used for representing an optimal corresponding relation of the time sequence feature data of the two time periods in a time dimension; Nonlinear time regular alignment is carried out on time sequence characteristic data in two time periods based on the optimal curved path to obtain characteristic point pairs; And calculating the feature distance between each feature point pair and carrying out accumulation and summation processing to obtain the feature matching distance.
5. The visual image-based upper limb movement intention recognition and trajectory prediction method according to claim 4, wherein the matching confidence degree is obtained by the following steps: normalizing the feature matching distance, and acquiring corresponding morphological similarity according to a pre-stored mapping relation, wherein the morphological similarity in the mapping relation is in negative correlation with the feature matching distance; Calculating an included angle cosine value between two time periods corresponding to instantaneous motion vectors to obtain a direction matching degree, wherein the instantaneous motion vectors are composed of the space speed and the angular speed of key points of the upper limbs; and carrying out weighted coupling processing on the morphological similarity and the direction matching degree to obtain the matching confidence degree.
6. The visual image-based upper limb movement intention recognition and trajectory prediction method according to claim 5, wherein the determining the upper limb movement intention of the target object in the current period based on the matching confidence, the time sequence feature data of the current period and the preset intention determination rule specifically comprises: the matching confidence coefficient is respectively compared with a first confidence coefficient threshold value and a second confidence coefficient threshold value, the upper limb movement intention of the current period is determined according to a preset intention judging rule, and the first confidence coefficient threshold value is larger than the second confidence coefficient threshold value; if the matching confidence coefficient is larger than a first confidence coefficient threshold value, determining that the upper limb movement intention in the current period is consistent with the upper limb movement intention in the previous period; if the matching confidence coefficient is smaller than the second confidence coefficient threshold value, the upper limb movement intention is redetermined based on the time sequence characteristic data of the current period; and if the matching confidence coefficient is positioned between the second confidence coefficient threshold value and the closed interval corresponding to the first confidence coefficient threshold value, executing correction operation on the upper limb movement intention determined in the previous period.
7. The visual image-based upper limb movement intention recognition and trajectory prediction method of claim 6, wherein the performing of the correction operation on the upper limb movement intention determined in the previous period of time specifically comprises: obtaining a predicted speed vector corresponding to the upper limb movement intention determined in the previous period, wherein the predicted speed vector consists of a movement direction and a movement rate corresponding to the upper limb movement intention in the previous period and is used for representing the expected movement state of the upper limb; acquiring a real-time speed vector corresponding to time sequence characteristic data of a current period, wherein the real-time speed vector is composed of an actual movement direction and a movement rate of the current period and is used for representing the current movement state of an upper limb; And carrying out weight configuration on the predicted speed vector and the real-time speed vector according to the matching confidence, and determining the result obtained by weighting and fusing the predicted speed vector and the real-time speed vector as the upper limb movement intention in the current period.
8. The visual image-based upper limb movement intention recognition and trajectory prediction method according to claim 1, wherein the preset upper limb movement constraint conditions comprise a geometric reachable boundary constraint condition for the upper limb key point space position in the initial prediction trajectory and a dynamic boundary constraint condition for the upper limb key point movement speed in the initial prediction trajectory; the geometric reachable boundary constraint condition is used for limiting the coordinate of the predicted point corresponding to the upper limb key point in the initial predicted track not to exceed the set reachable space range; The dynamic boundary constraint condition is used for limiting displacement difference values corresponding to key points of the upper limbs at adjacent moments in the initial prediction track to be not larger than a set maximum displacement threshold value.
9. The visual image-based upper limb movement intention recognition and track prediction method as set forth in claim 8, wherein the adjusting the initial predicted track and outputting the target predicted track specifically includes: Judging whether the coordinates of the predicted points of the upper limb key points in the initial predicted track meet the geometric reachable boundary constraint condition, if yes, taking the coordinates of the predicted points as intermediate coordinates, and if not, radially projecting the predicted points to the boundary of the reachable space range to obtain the intermediate coordinates; judging whether the intermediate coordinates meet the dynamic boundary constraint condition, if so, taking the intermediate coordinates as target predicted point coordinates, otherwise, taking the predicted point coordinates at the previous moment as the center, keeping the direction of the displacement vector unchanged, and cutting off the modular length of the displacement vector to be the maximum displacement threshold value to obtain the target predicted point coordinates; And connecting the coordinates of the target prediction points to output a target prediction track.
10. The upper limb movement intention recognition and track prediction system based on the visual image is applied to the upper limb movement intention recognition and track prediction method based on the visual image as claimed in any one of claims 1 to 9, and is characterized by comprising a visual image acquisition module, a time sequence feature extraction module, a matching confidence calculation module, an intention track prediction module and a movement constraint adjustment module; the visual image acquisition module is used for acquiring a visual image sequence of the upper limb movement process of the target object in the current period and extracting two-dimensional pixel coordinates of key points of the upper limb; The time sequence feature extraction module is used for obtaining pixel depth values of key points of the upper limbs, fusing the pixel depth values with two-dimensional pixel coordinates to obtain three-dimensional space position coordinates, extracting time sequence features of the upper limb movements based on the three-dimensional space position coordinates, and obtaining time sequence feature data used for representing the change features of the movement directions and the movement trend features of the upper limbs of the target object; The matching confidence calculation module is used for carrying out time sequence alignment and motion characteristic matching based on dynamic time warping on time sequence characteristic data of the current time period and the previous time period, and calculating the matching confidence between the upper limb motion intention determined in the previous time period and the upper limb motion track change in the current time period; the intention track prediction module is used for determining the upper limb movement intention of the target object in the current period based on the matching confidence degree, the time sequence characteristic data of the current period and a preset intention judgment rule, and predicting the upper limb movement track of the target object in the next period according to the upper limb movement intention of the current period to obtain an initial prediction track; The motion constraint adjustment module is used for judging whether the initial predicted track meets the preset upper limb motion constraint condition, if yes, outputting the initial predicted track as a target predicted track, and if not, outputting the target predicted track after adjusting the initial predicted track.

Description

Visual image-based upper limb movement intention recognition and track prediction method and system Technical Field The invention relates to the technical field of vision and pattern recognition, in particular to a method and a system for recognizing upper limb movement intention and predicting track based on a visual image. Background In the field of rehabilitation medicine, particularly in the upper limb rehabilitation assistance technology, an exoskeleton robot has been widely used as an assistance device to help a patient recover a motor function, particularly in a pathological state such as nerve injury or limb paralysis. The upper limb exoskeleton can not only provide the strength support required by the patient body, but also help the patient to exercise and train the limbs. However, in the rehabilitation process, how to accurately capture and identify the movement intention of the patient and predict the movement track so as to realize accurate man-machine coordination control is a key problem. In the prior art, the upper limb movement intention recognition and trajectory prediction are generally implemented by using continuous image or video data as input through a multi-stage processing flow of sensing high-level understanding by a lower layer. Firstly, visual information of the movement of the upper limbs of the human body is acquired, and the image is preprocessed and detected by the human body so as to reduce environmental interference. Upper limb keypoints or pose information is then extracted, converting the image data into a structured motion representation. And constructing time sequence movement characteristics on the basis, and modeling the dynamic change of the upper limb movement so as to identify the movement intention of an operator. Finally, the historical motion information and the identified intention are combined to predict the future motion trail of the upper limb, and the result is applied to scenes such as man-machine cooperation, rehabilitation assistance or intelligent interaction. A pedestrian recognition method and system based on image and motion behavior prediction is disclosed in China patent publication No. CN119810759B, and comprises the steps of collecting and analyzing environment and pedestrian information to obtain target environment and pedestrian information, predicting motion behavior tracks of pedestrians based on a GCN network and DeepSort algorithm in combination with the target environment and the pedestrian information to generate a pedestrian track prediction result, drawing up a user motion track based on an RRT algorithm and a PRM algorithm according to the pedestrian track prediction result, carrying out motion track simulation based on a multilayer LSTM model and a Seq2Seq model according to the pedestrian track prediction result and the user motion track to obtain a simulation result, and generating and sending safety warning information according to the simulation result. However, in the process of implementing the technical scheme of the embodiment of the application, the application discovers that the above technology has at least the following technical problems: Because the rehabilitation training object often has the problems of suffering limb muscular atrophy, left and right upper limb symmetry damage, abnormal posture and the like, the overall appearance and the outline of the upper limb deviate from the normal human body form obviously, so that the local features of key joint areas such as shoulders, elbows, wrists and the like in visual images are not obvious, stable and reliable key point coordinates are difficult to extract accurately, the key point detection confidence is reduced, and the positioning drift or loss phenomenon is easy to occur. Meanwhile, as the coordinate extraction errors of the key points are accumulated in the time dimension, the time sequence characteristics (such as joint angle change, speed and acceleration information) of the upper limb movement cannot be accurately described, so that the distinguishing capability of a movement mode is weakened, and the accuracy of the upper limb movement intention recognition is directly influenced. In addition, the unstable or distorted key point sequence can damage the learning of the track prediction model on the motion continuity and trend, so that the predicted track generates deviation in the spatial position and the motion direction, the actual motion intention and the future motion trend of a patient are difficult to truly reflect, and the accuracy of the upper limb motion intention recognition and the track prediction in the rehabilitation auxiliary training process is further reduced. Disclosure of Invention In order to solve the technical problems of inaccurate upper limb movement intention recognition and track prediction caused by unobvious upper limb local features and low key point detection confidence in a rehabilitation training visual image in the prior art, the e