CN-121982644-A - Human track prediction method and monitoring system for human-computer cooperation

CN121982644ACN 121982644 ACN121982644 ACN 121982644ACN-121982644-A

Abstract

The invention belongs to the technical field of human-computer cooperation interaction motion monitoring, and provides a human track prediction method and a monitoring system for human-computer cooperation, wherein the method comprises the steps of obtaining multi-view image data; based on the multi-view image data, reconstructing three-dimensional positions, linear speeds and angular speeds of a plurality of joint points of a human body and the tail ends of all connecting rods of a robot in real time to generate a motion state sequence, inputting three-dimensional coordinates of the joint points of the human body in the past T frames in the motion state sequence into a deep neural network model, generating multi-mode human body motion track prediction results of K future S frames in a single step, calculating potential danger indexes based on the multi-mode human body motion track prediction results, the current human-machine relative pose and the motion state, triggering a grading early warning signal according to a preset threshold, transmitting the grading early warning signal to a robot controller to execute deceleration, pause or avoidance actions, and synchronously displaying a human-machine digital twin body, a predicted track cloud and a risk grade on a three-dimensional visual interface.

Inventors

XU XIAOLONG
WANG XIN
SUN YUJIE
SONG YONG
Pang bao
XIA YIFAN
XU QINGYANG
YUAN XIANFENG

Assignees

山东大学

Dates

Publication Date: 20260505
Application Date: 20260128

Claims (10)

1. A human trajectory prediction method facing human-computer cooperation is characterized by comprising the following steps: capturing reflective mark points on an operator and a cooperative robot in real time, and acquiring multi-view image data; based on the multi-view image data, reconstructing three-dimensional positions, linear speeds and angular speeds of a plurality of joint points of a human body and tail ends of all connecting rods of the robot in real time, and generating a motion state sequence with update frequency not lower than a set value; Inputting three-dimensional coordinates of human body joint points of the past T frames in the motion state sequence to a depth neural network model trained by distillation, and generating multi-mode human body motion trail prediction results of K future S frames in a single step; calculating potential risk indexes based on the multi-mode human motion trail prediction result, the current human-machine relative pose and motion state, and triggering a grading early warning signal according to a preset threshold; and sending the grading early warning signals to a robot controller to execute deceleration, suspension or avoidance actions, and synchronously displaying a man-machine digital twin body, a predicted track cloud and a risk grade on a three-dimensional visual interface.
2. The human trajectory prediction method for human-computer collaboration according to claim 1, wherein the metric optical three-dimensional motion capture device is installed in a spatial triangle layout, microsecond frame synchronization is achieved through hardware triggering, and the reflective marker points are respectively arranged on the shoulder, elbow, wrist, hip, knee, ankle and head of a human body, and the base, shoulder joint, elbow joint, wrist joint and end effector of the collaborative robot.
3. The human-computer collaboration-oriented human track prediction method as claimed in claim 1, wherein the method is characterized in that based on the multi-view image data, three-dimensional positions, linear velocities and angular velocities of a plurality of joints of a human body and tail ends of connecting rods of a robot are reconstructed in real time by adopting an incremental quaternion fusion algorithm, and specifically comprises the following steps: applying an ergonomic topological constraint to the reconstructed 3D mark points, and solving a joint rotation sequence expressed by unit quaternions; calculating the accurate pose of each connecting rod under the global coordinate system by spherical linear interpolation on the basis of the parameter model and encoder feedback of the cooperative robot; And fusing the pose data of the unmanned aerial vehicle, and outputting a three-dimensional motion state vector with millimeter-level precision.
4. The human-computer collaboration-oriented human trajectory prediction method of claim 1, wherein the training process of the deep neural network model comprises: Firstly training a teacher model solved based on an ordinary differential equation ODE, and learning multi-modal future track distribution through a Flow Matching mechanism; and training a lightweight chemical raw model by adopting a IMLE distillation method, so that the lightweight chemical raw model can be sampled in a single step to generate K tracks consistent with the distribution of a teacher model.
5. The human trajectory prediction method for human-computer collaboration according to claim 1, wherein the potential risk index PDI is calculated according to the following formula: Wherein, the Euclidean distance between closest points of the human machine, As a projection component of the relative velocity in the man-machine connection direction, Is the physical and cardiac acceleration of a human body, Is a preset weight.
6. The human-computer collaboration-oriented human trajectory prediction method of claim 1, wherein when PDI (t) > A yellow alarm is triggered when PDI (t) > Or triggering red early warning when the intersection probability of any predicted track and the robot working space is larger than a set value, wherein 。
7. A human track monitoring system facing human-computer cooperation is characterized by comprising the following steps: Metric optical three-dimensional motion capture device and processor; the measuring optical three-dimensional motion capture device is used for capturing reflective mark points on an operator and the cooperative robot in real time and acquiring multi-view image data; the processor is configured to include: The three-dimensional reconstruction and pose estimation module is configured to reconstruct three-dimensional positions, linear speeds and angular speeds of a plurality of joint points of a human body and tail ends of all connecting rods of the robot in real time based on the multi-view image data, and generate a motion state sequence with update frequency not lower than a set value; The behavior recognition and track prediction module is configured to input three-dimensional coordinates of human body joint points of the past T frames in the motion state sequence to a deep neural network model trained by distillation, and generate multi-mode human body motion track prediction results of K future S frames in a single step; the risk assessment and early warning module is configured to calculate potential risk indexes based on the multi-mode human motion trail prediction result, the current human-machine relative pose and motion state, and trigger a grading early warning signal according to a preset threshold; and the self-adaptive control and visualization module is configured to send the grading early warning signal to a robot controller to execute deceleration, suspension or avoidance actions, and synchronously display a man-machine digital twin body, a predicted track cloud and a risk grade on a three-dimensional visualization interface.
8. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the method of any one of claims 1 to 6.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of the preceding claims 1-6 when the program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, performs the steps of the method of any of the preceding claims 1-6.

Description

Human track prediction method and monitoring system for human-computer cooperation Technical Field The invention belongs to the technical field of human-computer cooperation interaction motion monitoring, and particularly relates to a human track prediction method and a human track prediction system for human-computer cooperation. Background The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art. With the rapid development of collaborative robot technology, a scene of collaborative operation of robots and human beings in the same working space is becoming more common. However, in practical applications, if an effective safety protection mechanism is lacking, accidental contact or collision between the robot and the human body may still cause personal injury. The current safety evaluation method is mostly dependent on an offline theoretical model or experimental data under a limited working condition, and is difficult to realize real-time, dynamic and active safety evaluation of a human-computer interaction process. In the prior art, the monitoring means aiming at human-computer interaction safety mainly comprises a force/moment sensor, a collision detection algorithm, safety area setting and the like. For example, a pressure sensor, an accelerometer, etc. are installed at the robot joint or end for triggering a sudden stop after physical contact occurs. Although the method can lighten collision results to a certain extent, the method essentially belongs to a passive response type safety mechanism, has obvious limitations, including depending on sensing after contact, being incapable of early warning before collision occurs, being difficult to adapt to high-speed unstructured dynamic interaction based on static or quasi-static assumptions, being easily influenced by vision blind areas or sensor shielding in multi-person and multi-shielding industrial environments, causing false alarm or missing alarm, having higher overall response delay of the system, generally being in the order of hundreds of milliseconds, and being incapable of meeting the requirement of standards such as ISO/TS 15066 and the like on millisecond-level safety response of the cooperative robot. In addition, conventional industrial robot systems typically employ a man-machine isolation strategy, i.e., a robot that prohibits personnel from entering its work area while running. However, in a large number of practical application scenarios, such as flexible assembly, customized production, equipment maintenance, etc., tasks must rely on manual intervention and cannot be fully automatically completed by the robot. The scene requires continuous interaction between human and robot under the conditions of close distance, high dynamic and random behavior, and has higher requirements on a safety monitoring system. In recent years, partial researches try to introduce a deep learning method to improve the safety of human-computer interaction, in particular to a human motion trail prediction technology based on a neural network. However, the existing methods still have the following key problems: In the prior art, although part of generated models such as MID, LED and the like based on a diffusion mechanism can output multi-mode prediction tracks, the denoising and sampling process usually needs hundreds to thousands of steps of iteration, the single reasoning time is more than 500 milliseconds, and the real-time threshold acceptable for a far-beyond man-machine cooperation scene is usually less than 50ms. The existing mainstream track prediction model is mostly trained and verified on pure pedestrian interaction data sets such as ETH, UCY and the like, and the special cautiousness, avoidance intention or task guiding behavior of human being when facing the mechanical arm is not considered, so that the generalization capability of the model in a real human-robot co-fusion environment is obviously reduced. The existing deterministic prediction methods such as LSTM, the early version of the Social-GAN and the like only output a single future track, cannot characterize inherent multi-mode and uncertainty of human movement, and are difficult to support reliable risk assessment. Disclosure of Invention In order to overcome the defects in the prior art, the invention provides a human track prediction method and a human track monitoring system for human-computer cooperation, which can predict the track of a pedestrian and monitor the relative distance between a robot and a human body in real time so as to ensure the safety of human-computer cooperation. To achieve the above object, one or more embodiments of the present invention provide the following technical solutions: in a first aspect, a human trajectory prediction method for human-computer collaboration is disclosed, including: capturing reflective mark points on an operator and a cooperative robot in real time,