CN-121999535-A - Motion attitude anomaly real-time detection and correction method based on space-time convolution network

CN121999535ACN 121999535 ACN121999535 ACN 121999535ACN-121999535-A

Abstract

The invention discloses a motion gesture abnormality real-time detection and correction method based on a space-time convolution network, and relates to the field of gesture detection. And extracting skeleton key points of a human body, and adjusting the skeleton key points according to the skeleton of the human body through a skeleton extraction algorithm, so that the skeleton key points for detection are more accurate. The natural connection mode of the human body is detected to connect the bone key. And (3) extracting the space feature map by constructing an adjacency matrix and adopting a map convolution mode. Detecting the positions of the same skeleton key point in a plurality of time points to construct a three-dimensional image so as to judge the position change of the skeleton key point at the plurality of time points, and judging whether the gesture is abnormal or not by combining the change state of the characteristics in the space characteristic diagram along with the time. The technical effect of enabling the gesture judgment to be more accurate is achieved.

Inventors

LIU CHENHU
DING LEI
GUO YANHUA
WANG CHUNYANG
SUN LIJIE
ZHANG RUIPING
SUN LI

Assignees

北京一石科技有限责任公司

Dates

Publication Date: 20260508
Application Date: 20260212

Claims (10)

1. A motion attitude anomaly real-time detection and correction method based on a space-time convolution network is characterized by comprising the following steps: Acquiring moving images of a plurality of time points, wherein the moving images are images of a human body containing movement to be detected; Based on the moving image, respectively detecting a skeleton and skeleton key points to obtain skeleton key point images and an adjacent matrix, wherein the skeleton key point images comprise a plurality of skeleton detection points adjusted by the skeleton, and the adjacent matrix represents the connection relation of the stored skeleton key points; Based on the skeleton key point image and the adjacent matrix, performing space detection to obtain a space feature map; performing time detection based on the skeleton key point image to obtain a track image; based on the space feature map and the plurality of track images, judging the posture change which changes along with time to obtain a time feature map; And judging whether the gesture is abnormal or not according to the time feature diagram.
2. The method for detecting and correcting motion gesture anomalies in real time based on a space-time convolution network according to claim 1, wherein the steps of respectively detecting a skeleton and a skeleton key point based on the motion image to obtain a skeleton key point image include: obtaining skeleton key points according to the moving image, wherein the skeleton key points represent positions on the skeleton, which have influence on detection; connecting skeleton key points according to the moving image to obtain a first key point image, wherein the first key point image comprises a plurality of skeleton key points and skeleton line segments, and the skeleton line segments represent line segments connecting the skeleton key points; By constructing a graph structure, storing the connection relation of the skeleton key points to obtain an adjacent matrix; And obtaining skeleton key point images and skeleton images based on the adjacent matrix and the moving image through skeleton extraction, wherein the skeleton key point images are adjusted first key point images.
3. The method for detecting and correcting motion gesture anomalies based on a space-time convolution network according to claim 2, wherein the obtaining skeleton key point images and skeleton images based on the adjacency matrix and the motion images through skeleton extraction comprises: dividing a human body in the moving image to obtain a plurality of local human body images, wherein 1 local human body image corresponds to 1 skeleton line segment; extracting the skeleton of the local human body image to obtain a first skeleton image; Fitting a plurality of points in a plurality of first skeleton images into straight lines serving as skeleton straight lines to obtain skeleton images, wherein the skeleton images comprise 1 skeleton straight line; according to the adjacent matrix, acquiring a skeleton line segment communicated with skeleton key points, fitting the skeleton line segment into a straight line, and obtaining a communicated skeleton straight line; taking the central points of a plurality of polygons which are communicated with the skeleton straight lines and the corresponding skeleton straight lines as adjusted skeleton key points; And connecting the adjusted skeleton key points according to the adjacency matrix to obtain skeleton key point images, wherein skeleton line segments in the skeleton key point images are line segments connected with the adjusted skeleton key points.
4. The method for detecting and correcting motion gesture anomalies based on a space-time convolution network according to claim 1, wherein the performing time detection based on the skeletal key point images to obtain a plurality of track images comprises: Dividing according to skeleton line segments in the skeleton key point images to obtain a plurality of local skeleton images; Detecting a skeleton line segment midpoint corresponding to the local skeleton image to obtain a skeleton midpoint; connecting the middle points of the skeletons according to a plurality of time points to obtain track images; The long of the track image represents the time point, and the wide and high represent the positions of the framework midpoints in the two-dimensional space.
5. The method for detecting and correcting motion gesture anomalies in real time based on a space-time convolution network according to claim 1, wherein the determining gesture changes over time based on the spatial feature map and the plurality of trajectory images to obtain a temporal feature map comprises: In the space feature map, inputting features corresponding to the same skeleton key points into a time convolution network, and detecting the change of the skeleton key points along with time to obtain an overall time feature map; inputting the track images into a three-dimensional convolutional neural network, and judging the local motion change of the human body to obtain local time feature vectors; and fusing the whole time feature vector and the plurality of local time feature vectors to obtain a time feature map.
6. The method for detecting and correcting motion gesture anomalies in real time based on a space-time convolutional network according to claim 5, wherein the convolution kernel corresponding to the three-dimensional convolutional neural network is 2 n M, 2 corresponds to a time point, n represents the length of the track image, and m represents the width of the track image.
7. The method for detecting and correcting motion gesture anomalies in real time based on a space-time convolution network according to claim 1, wherein the step of performing spatial detection based on the skeleton key point image and an adjacent matrix to obtain a spatial feature map comprises the steps of: Taking 1 skeleton key point in the skeleton key point image as a root node according to the adjacency matrix; taking other skeleton key points with connection relation at the root node as child nodes; constructing a skeleton key set by 1 root node and a plurality of child nodes; based on the skeleton key point image, judging the position difference through the skeleton key set to obtain a three-dimensional key point image; And detecting the attitude change based on the three-dimensional key point image through a convolution kernel to obtain a space feature map.
8. The method for detecting and correcting motion gesture anomalies based on a space-time convolution network according to claim 7, wherein the step of determining a position difference based on the skeleton key point image through the skeleton key set and the gravity center position to obtain a spatial feature map comprises the steps of: the gravity center position is obtained and represents the average position of all the joint points; calculating distances between a plurality of bone key points in the bone key set and the gravity center position as gravity center distances; Constructing a root node adjacency matrix by using skeleton key points corresponding to the root nodes; constructing a centripetal adjacency matrix by using skeleton key points smaller than the gravity center distance corresponding to the root node; constructing a centrifugal adjacency matrix by using skeleton key points larger than the gravity center distance corresponding to the root node; and convolving based on the root node adjacency matrix, the centripetal adjacency matrix and the centrifugal adjacency matrix to obtain a space feature map.
9. The method for detecting and correcting motion gesture anomalies in real time based on a space-time convolution network according to claim 8, wherein the convolving based on a root node adjacency matrix, a centripetal adjacency matrix and a centrifugal adjacency matrix to obtain a spatial feature map comprises: constructing a bone tensor according to the positions of the bone key points, the time points and the number of the bone key points; respectively carrying out graph convolution according to the bone tensor, the root node adjacency matrix, the centripetal adjacency matrix and the centrifugal adjacency matrix to obtain three first space feature graphs; And combining the three first space feature images to obtain a first space feature image.
10. The method for detecting and correcting abnormal motion gesture based on space-time convolution network according to claim 1, wherein the step of determining whether the gesture is abnormal according to the time feature map comprises the steps of: acquiring a plurality of normal feature images, wherein the normal feature images are feature images acquired by normal postures; Calculating similarity between the normal feature map and the time feature map through a cosine similarity algorithm to obtain feature similarity values; If the feature similarity value is larger than the similarity threshold value, setting the posture as normal; if all the feature similarity values are smaller than the similarity threshold value, the gesture is set to be abnormal.

Description

Motion attitude anomaly real-time detection and correction method based on space-time convolution network Technical Field The invention relates to the field of gesture detection, in particular to a motion gesture abnormality real-time detection and correction method based on a space-time convolution network. Background In many areas, such as industrial manufacturing, sports, medical rehabilitation, etc., incorrect exercise attitudes may lead to injuries. Real-time detection and correction can prevent these injuries. Proper posture helps to improve athletic performance and training. The real-time feedback can help people adjust actions in time, so that correct skills can be mastered more quickly. For the elderly or patients, abnormal exercise postures can be signs of health problems, and real-time monitoring can timely discover the abnormalities and remind of seeking medical attention or adjust the activity mode. In industrial manufacturing, whether or not the operating posture of staff meets specifications directly relates to production safety and product quality. Real-time detection may ensure that the operating specification is being executed. In Virtual Reality (VR) and Augmented Reality (AR) applications, real-time gesture correction can promote user experience, making interactions more natural and accurate. Traditional CNNs are adept at processing euclidean data (e.g. images, regular grids), but the skeletal joints of the human body naturally constitute a graph structure (skeletal key points are nodes, and skeletal edges). At present, whether the gesture is abnormal or not can be judged through static gesture abnormality, but the abnormality is judged through continuous integral gesture in the motion process, and the motion trail is more in line with the motion condition. But how to accurately judge the pose is a problem. Disclosure of Invention The invention aims to provide a motion gesture abnormality real-time detection and correction method based on a space-time convolution network, which is used for solving the problems in the prior art. In a first aspect, an embodiment of the present invention provides a method for detecting and correcting motion gesture anomalies in real time based on a space-time convolution network, including: Acquiring moving images of a plurality of time points, wherein the moving images are images of a human body containing movement to be detected; Based on the moving image, respectively detecting a skeleton and skeleton key points to obtain skeleton key point images and an adjacent matrix, wherein the skeleton key point images comprise a plurality of skeleton detection points adjusted by the skeleton, and the adjacent matrix represents the connection relation of the stored skeleton key points; Based on the skeleton key point image and the adjacent matrix, performing space detection to obtain a space feature map; performing time detection based on the skeleton key point image to obtain a track image; based on the space feature map and the plurality of track images, judging the posture change which changes along with time to obtain a time feature map; And judging whether the gesture is abnormal or not according to the time feature diagram. Optionally, based on the moving image, respectively performing skeleton and skeleton key point detection to obtain a skeleton key point image, including: obtaining skeleton key points according to the moving image, wherein the skeleton key points represent positions on the skeleton, which have influence on detection; connecting skeleton key points according to the moving image to obtain a first key point image, wherein the first key point image comprises a plurality of skeleton key points and skeleton line segments, and the skeleton line segments represent line segments connecting the skeleton key points; By constructing a graph structure, storing the connection relation of the skeleton key points to obtain an adjacent matrix; And obtaining skeleton key point images and skeleton images based on the adjacent matrix and the moving image through skeleton extraction, wherein the skeleton key point images are adjusted first key point images. Optionally, the skeleton extraction, based on the adjacency matrix and the moving image, obtains a skeleton key point image and a skeleton image, including: dividing a human body in the moving image to obtain a plurality of local human body images, wherein 1 local human body image corresponds to 1 skeleton line segment; extracting the skeleton of the local human body image to obtain a first skeleton image; Fitting a plurality of points in a plurality of first skeleton images into straight lines serving as skeleton straight lines to obtain skeleton images, wherein the skeleton images comprise 1 skeleton straight line; according to the adjacent matrix, acquiring a skeleton line segment communicated with skeleton key points, fitting the skeleton line segment into a straight line, and obtaining a communicated s