CN-118081740-B - Robot intelligent article transfer method and system oriented to man-machine cooperation

CN118081740BCN 118081740 BCN118081740 BCN 118081740BCN-118081740-B

Abstract

The invention discloses a robot intelligent article transfer method and a system for man-machine cooperation, which belong to intelligent article transfer between a robot and a person, wherein the method comprises the steps of constructing a manipulator with proximity sensing and pressure sensing, fusing vision to perform multi-mode sensing, and sensing the proximity of the manipulator in real time; the method comprises the steps of constructing a human and robot digital twin system in the object transfer process based on virtual bounding boxes of objects, human hands and robots, evaluating the arrival state of the human hands in real time, collecting data by taking the positions and postures of the positions from a receiver to the transfer points and the time when the transfer points are completed by the transfer points as core indexes in the transfer process of different objects of the human and the human, constructing a data set, developing a reinforcement learning algorithm to train out a transfer time selection model of the human, and enabling the robot to have the capability of actively selecting the transfer time.

Inventors

ZHANG SHUAI
TAO FEI
ZUO YING
ZHANG HE
ZHOU YAOMING

Assignees

天目山实验室
北京航空航天大学

Dates

Publication Date: 20260508
Application Date: 20240131

Claims (5)

1. The intelligent article transfer method for the robot facing the man-machine cooperation is characterized by comprising the following steps of: S1, constructing a manipulator with proximity sensing and pressure sensing, fusing vision to perform multi-mode sensing, and sensing the proximity of a human hand in real time; S2, constructing a human and robot digital twin system in the process of transferring objects based on virtual bounding boxes of the objects, the human hands and the robots, wherein the digital twin system can evaluate the arrival state of the human hands in real time based on the visually updated human hand position, the robot kinematically updated grabbed object position and the space geometrical relationship between the two positions; S3, collecting data according to the pose of a receiver to a transfer point and the transfer time of a transfer person, namely the time of loosening an object, in the transfer process of different articles of a person-person, wherein the core index data comprises three core indexes, namely the position of a human hand based on visual calculation, the approaching state of the human hand based on variable sensing of capacitance and the stay time of the human hand at the position, constructing a dataset based on the position, developing an improved DDPG algorithm based on a depth certainty strategy gradient DDPG, and training a transfer time selection model of the person-like by using the constructed dataset, wherein the model can enable a robot to have the capability of actively selecting the transfer time; in step S3, an intelligent object transfer algorithm is realized by using a hand pose recognition module, a DDPG module and a robot control module; The method comprises the steps of utilizing a human hand pose recognition module to recognize the pose of a receiver when reaching the stop of the position of a grabbing point, starting timing at the moment when the hand of the receiver stops moving, recording the time when a transmitter releases an object, calculating the stay time of the human hand of the receiver at the position, and constructing a data set; the DDPG module contains a modified DDPG algorithm and a prize value function, wherein the modified DDPG algorithm includes the steps of: Firstly, describing a man-machine intelligent article transfer problem as a Markov decision problem, setting an action executed by each time step intelligent agent as a t , and corresponding states as follows: Wherein θ is the current joint angle of the robot, d is the minimum distance between the human hand and each joint of the robot, d min is the minimum value of the minimum distances d between the human hand and each joint of the robot in the current state, K i is the joint number corresponding to d min , P is the end position of the mechanical arm and comprises coordinate values of x, y and z, Q is the end gesture of the robot and is expressed in a quaternion mode of x, y, z and w; next, a prize value function is constructed, the actions are The prize value R k can be solved by the following formula: Wherein, gamma epsilon [0,1] is an attenuation factor, and the return of the kth step length is as follows: wherein s is the current time state, P 0 is the terminal pose when the mechanical arm reaches the transmission state; The terminal pose of the hand in the current state is represented, t 0 represents the moment when the robot reaches the transmission position, Representing the current moment, wherein m e and m k respectively represent the positions of the tail ends of the hands and the proportion of the transmission time in a reward function, and in the process of constructing a data set by using the reward value function, a receiver scores the tail end pose of the hands and the transmission time of the robot according to a given Like special score amount table, and the scores are quantized to obtain values of m e and m k ; the improved DDPG algorithm strengthens the receiver gesture and the transmission opportunity, and as the reward value function is the score of human beings on the transmission opportunity under different transmission gestures, the transmission opportunity selection model obtained by training gradually converges to the transmission opportunity with the highest score; When the robot control module invokes the transfer opportunity selection model to control the article transfer process of the robot, the robot control module combines the approach perception of multi-mode fusion, takes the reached hand gesture as input after completing the state evaluation, generates the optimal control time for opening the robot clamp, and realizes the active article transfer of the robot.
2. The method of claim 1, wherein S1 comprises selecting a suitable proximity capacitance plate material and a piezoelectric material, designing a manipulator to cover a finger of a hand with the proximity capacitance plate material, attaching the piezoelectric material to the surface of the capacitance plate material as a pressure sensor, changing the proximity capacitance by means of human hand proximity, and realizing real-time and multi-modal sensing of the proximity state of the human hand in combination with a visual sensor.
3. The method according to claim 2, wherein the step S1 comprises the steps of taking a metal material as a manipulator structure coated by a proximity capacitance sensing polar plate material, arranging a visual sensor, and fusing a multi-mode human hand proximity sensing algorithm through capacitance change and a machine visual algorithm; In the multi-mode hand proximity sensing algorithm, solving the distance between a hand and a robot hand according to the following formula, wherein d refers to the solved distance between the hand and the robot hand, d 1 refers to the distance between the hand calculated based on a visual algorithm, and d 2 refers to the distance between the hand based on a proximity capacitance test; in the vision algorithm, the formula is adopted Wherein traA (t) and traB (t) are the trajectories of a hand A and an object or a clamp B in a time period t [ t0, t1] respectively, solving the real-time distance between the hand and the manipulator, and based on the change of the proximity capacitance C, referencing the equation relationship Wherein And d 2 is solved by taking the dielectric constant, k as the electrostatic force constant and S as the area of the cladding polar plate.
4. The method according to claim 3, wherein the step S2 comprises constructing a virtual bounding box of a human hand, a gripped object and a mechanical arm comprising a mechanical arm, and establishing a spatial relationship between the gripped object and each joint of the mechanical arm according to the positive kinematics of the mechanical arm, so as to construct a digital twin system of the human hand, the gripped object and the mechanical arm, wherein a calculation formula is the same as a calculation formula of d 1 in the step S1, a distance relationship between the bounding box of the human hand and the gripped object is solved, and a human hand arrival state is estimated according to the distance relationship between the gripped object and the human hand in a digital space.
5. A transfer system for a robot intelligent object transfer method for human-machine collaboration according to any one of claims 1-4, wherein the system comprises a hierarchical human hand proximity sensing module, a system digital twin module, an active transfer instruction module, a robot control module and a data acquisition module, wherein, The hierarchical human hand approach sensing module is used for performing multi-mode implementation sensing on the approach state of the human hand in the process of transferring the articles between the robot and the human; The system digital twin module is used for evaluating the arrival state of hands and providing materials for training the model selected at the transmission time; the active transmission instruction module is used for outputting a transmission trigger node instruction according to the input hand gesture; the robot control module is used for controlling the robot to move to the contact point and triggering the node to complete the transmission task according to the output transmission instruction; and the data acquisition module is used for providing experience data of the transfer time of the person and the person-to-person article in the neural network training and providing quantized data for constructing the reward value function.

Description

Robot intelligent article transfer method and system oriented to man-machine cooperation Technical Field The invention belongs to the technical field of robot man-machine cooperation, and particularly relates to a robot intelligent article transfer method and system for man-machine cooperation. Background With the development of collaborative robot technology, more and more robots will participate in our daily lives and production. In a human-computer cooperation task, the object transfer of a robot and a person is an important link in the human-computer cooperation process. How to develop and design a reasonable end effector, select a proper sensing channel, develop a robust grabbing state and a human intention sensing algorithm, and have important significance for guaranteeing the efficiency of the article transfer of the robot and the human in the cooperation process and improving the user experience of the robot. Item transfer from person to person can be divided into three processes, approach, arrival, and transfer. However, the current transfer of items to a person by a robot is mostly focused on the transfer process, and vision-based or force-sense-based methods are generally used to evaluate whether the robot has completed the transfer process. And the robot is usually triggered to open the clamping jaw passively after the preset fixed time is adopted or the force sense and the voice prompt of the person are received, the passive transmission mode is not consistent with the multi-mode-based inferred active transmission mode between the people, and the user experience is greatly reduced. Therefore, it is necessary to build a robot intelligent object transfer algorithm oriented to man-machine cooperation, and the system is based on the robot autonomous transfer control algorithm comprising multi-mode human hand proximity perception, comprising arrival state evaluation based on human hand and object digital twinning, comprising data sets of different object people and human transfer time, and reinforcement learning. By designing the end effector of the robot with intelligent proximity sensing capability, the robot is enabled to have the proximity and arrival of a person and the autonomous control of transmission by means of arrival state evaluation based on digital twin and space relativity and transmission time dynamic prediction based on reinforcement learning, and finally the aim of improving interaction experience of the cooperative robot is achieved. Attempts have been made to use robots to structurally initiate lifting of their ability to transfer items, e.g. application publication no The patent application CN111301838a discloses an intelligent robot for transferring items. However, this approach focuses only on the robot structure body, and it is difficult for the robot to actively sense the approach, arrival, and transfer states of the person. In order to estimate the approach of a human hand, the application publication number CN113681565A discloses a man-machine cooperation method and a device for realizing the object transfer between a robot and a human, and the object transfer is realized by controlling the movement of a mechanical arm to the position of the human hand in a mode of visually identifying the pose of the human hand. The method focuses more on the approach process in the transfer process, and the person belongs to the passive waiting role in the transfer process. In order to improve user experience, a more comprehensive article transfer algorithm and system are required to be designed so that the robot has the capability of actively selecting transfer time. Disclosure of Invention In view of the above, the invention aims to provide a robot intelligent article transfer method and a robot intelligent article transfer system for human-machine cooperation, which can realize quick perception of human hand approach, accurate estimation of human hand arrival transfer points and active selection of transfer opportunities, ensure robustness of a robot transfer process, and further improve interaction experience of the robot. To achieve the above object, according to one aspect of the present invention, there is provided a robot intelligent article transfer method for human-machine cooperation, the method comprising the steps of: S1, constructing a manipulator with proximity sensing and pressure sensing, fusing vision to perform multi-mode sensing, and sensing the proximity of a human hand in real time; S2, constructing a human and robot digital twin system in the process of transferring objects based on virtual bounding boxes of the objects, the human hands and the robots, wherein the digital twin system can evaluate the arrival state of the human hands in real time based on the visually updated human hand position, the robot kinematically updated grabbed object position and the space geometrical relationship between the two positions; S3, collecting data according to the pose of a rec