Search

CN-122008225-A - Robot grabbing system and method based on computer vision

CN122008225ACN 122008225 ACN122008225 ACN 122008225ACN-122008225-A

Abstract

The invention discloses a robot grabbing system and a robot grabbing method based on computer vision, and relates to the technical field of industrial vision intelligence and robot control. The method comprises the steps of obtaining a high-speed visual image sequence and multi-source disturbance data of the tail end part of a flexible target object in real time, extracting space-time characteristics of the image sequence to obtain tail end displacement space-time characteristics, constructing a nonlinear dynamic model based on the multi-source disturbance data and the displacement space-time characteristics, taking the nonlinear dynamic model as a prediction model, calculating an optimal control compensation sequence by solving a constraint optimization problem in a limited time domain, extracting a current moment control compensation generation instruction, driving a robot to adjust a grabbing track, and repeating the steps to realize rolling optimization control. According to the invention, the dynamic behavior of the flexible body is accurately described through the nonlinear dynamic model, and the active compensation of multi-source disturbance is realized by combining model predictive control, so that the technical problems of high-precision and high-stability grabbing of the flexible target in a complex environment are effectively solved.

Inventors

  • LUO DEYUN
  • LUO YUJIE

Assignees

  • 南京信息职业技术学院

Dates

Publication Date
20260512
Application Date
20260320

Claims (10)

  1. 1. The robot grabbing method based on computer vision is characterized by comprising the following steps of: Acquiring a high-speed visual image sequence of the tail end part of the flexible target object acquired by a visual sensor in real time, and acquiring multi-source disturbance data reflecting the environment and the state of the robot system; extracting space-time characteristics of the high-speed visual image sequence to obtain the space-time characteristics of the tail end displacement of the tail end part in a preset history window before the current moment; based on the multisource disturbance data and the terminal displacement space-time characteristics, constructing a nonlinear dynamic model describing dynamic behaviors of the flexible target object under the action of external excitation; Taking the nonlinear dynamic model as a prediction model, combining the actual displacement state of the tail end part at the current moment, and calculating to obtain an optimal control compensation sequence for compensating nonlinear displacement offset caused by multi-source disturbance by solving a constraint optimization problem in a finite time domain; Extracting control compensation at the current moment from the optimal control compensation sequence, generating a control instruction and sending the control instruction to a motion controller of the robot so as to drive an end effector of the robot to adjust a grabbing track; And repeatedly executing the steps at the next moment to realize the rolling optimization control.
  2. 2. The computer vision based robotic grasping method of claim 1, wherein the multi-source disturbance data comprises robot joint moment data collected by a robot joint moment sensor and environmental vibration data of a grasping station collected by an environmental vibration sensor; The robot grabbing method based on computer vision further comprises the following steps: Performing spectrum analysis on the robot joint moment data to obtain a deterministic disturbance sequence caused by robot motion; Performing time-frequency analysis on the environmental vibration data to obtain a periodic disturbance sequence caused by environmental vibration; the deterministic perturbation sequence and the periodic perturbation sequence are used as external excitation input of the nonlinear dynamic model.
  3. 3. The robot gripping method based on computer vision according to claim 1, wherein the extracting of the spatiotemporal features of the high-speed visual image sequence to obtain the end displacement spatiotemporal features of the end portion in a preset history window before the current time specifically comprises: Preprocessing and target segmentation are carried out on each frame of image in the high-speed visual image sequence, and a region of interest of the tail end part is extracted; And carrying out space-time feature extraction on the image sequence of the region of interest in the preset history window by adopting a three-dimensional convolution neural network to generate an end displacement space-time feature tensor containing space position information and time evolution information, wherein the convolution kernel of the three-dimensional convolution neural network slides along the space dimension and the time dimension of the image at the same time so as to capture the space-time correlation of the end part in continuous motion.
  4. 4. The robot gripping method based on computer vision according to claim 2, wherein constructing a nonlinear dynamic model describing the dynamic behavior of the flexible target object under the action of external excitation specifically comprises: combining the deterministic disturbance sequence and the periodic disturbance sequence into an external excitation vector The terminal displacement space-time characteristic is subjected to dimension reduction processing and then used as a system output vector ; A nonlinear autoregressive moving average model structure is adopted: Wherein the method comprises the steps of As a function of the non-linearity, 、 、 For the order of the model, Is modeling error; Training based on historical data by adopting a recurrent neural network or a nonlinear state space identification algorithm to obtain the nonlinear function And obtaining the nonlinear dynamic model.
  5. 5. The robot gripping method based on computer vision according to claim 1, wherein solving the constrained optimization problem in the finite field, and calculating to obtain an optimal control compensation sequence, comprises: setting an objective function of the optimization problem as follows: Wherein the method comprises the steps of To be at the moment Predicted The end part of the moment is displaced, For the reference displacement for which a grabbing trajectory is desired, The delta is compensated for the control to be solved, In order to predict the length of the time domain, In order to control the length of the time domain, And Is a weight matrix; Taking robot kinematics and dynamics constraint as constraint conditions of an optimization problem; Solving the optimization problem by adopting a real-time iterative optimization algorithm to obtain an optimal control compensation sequence 。
  6. 6. The computer vision based robotic grasping method of claim 5, wherein the constraint condition comprises a stability indicator of an end effector grasping gesture, the stability indicator calculated by: Based on the predicted displacement of the end region in the predicted time domain Calculating a corresponding robot joint angle and an end effector posture through inverse kinematics of the robot; Calculating a force spiral space at the contact point of the end effector and the flexible target object, judging whether a group of contact force can balance any external disturbance, if so, judging that the force is closed, and setting the stability index as a force closing measurement value; And adding the stability index as a soft constraint or a hard constraint into an objective function or constraint condition of the optimization problem.
  7. 7. The computer vision based robotic grasping method of claim 1, further comprising the step of on-line adaptive updating of the nonlinear dynamic model: recording in real time the actual displacement output at each instant And an external excitation input Forming an online dataset; Adopting a recursion extended least square method or a gradient-based online learning algorithm, and carrying out recursion correction on parameters of the nonlinear dynamic model by utilizing the online data set; And injecting the updated model parameters into the nonlinear dynamic model in real time for prediction and optimization at the subsequent moment.
  8. 8. The computer vision-based robotic grasping method of claim 1, wherein the method further comprises a real-time correction step based on visual feedback: Receiving a high-speed visual image at the current moment acquired by the visual sensor in real time, and extracting the actual three-dimensional space coordinate of the tail end part at the current moment; Comparing the actual three-dimensional space coordinate with a reference coordinate corresponding to the expected grabbing track, and calculating to obtain a real-time displacement deviation; and inputting the real-time displacement deviation into a PID controller, calculating by the PID controller to obtain a real-time correction instruction, and sending the real-time correction instruction and the control instruction to the motion controller after overlapping.
  9. 9. A computer vision-based robotic grasping system, comprising: The visual sensor is arranged near the end effector of the robot and is used for acquiring a high-speed visual image sequence of the end part of the flexible target object; The sensor group is used for collecting multi-source disturbance data reflecting the environment where the robot system is and the state of the robot system; a processor in communication with the vision sensor, the sensor set, and a motion controller of the robot, respectively, the processor configured to perform the computer vision-based robotic grasping method of any one of claims 1-8; and the motion controller is in communication connection with the processor and the robot body, and is used for receiving the control instruction generated by the processor and driving the robot end effector to adjust the grabbing track according to the control instruction.
  10. 10. The robotic grasping system based on computer vision according to claim 9, wherein the sensor group includes a robot joint moment sensor integrated inside each joint of the robot and an environmental vibration sensor provided on a base of the grasping station, the processor further includes a model predictive control engine and an on-line recognition module, wherein: the model predictive control engine is used for executing the step of solving the constrained optimization problem in the limited domain and generating an optimal control compensation sequence; The online identification module is used for executing the step of online self-adaptive updating of the nonlinear dynamic model; The processor is further configured with a high-speed data buffer for storing the end displacement space-time characteristics and the external excitation sequences in the preset history window so as to support real-time rolling optimization calculation of the model predictive control engine.

Description

Robot grabbing system and method based on computer vision Technical Field The invention relates to the technical field of industrial vision intelligence and robot control, in particular to a robot grabbing system and method based on computer vision. Background With the continuous improvement of the industrial automation level, the robot grabbing technology is widely applied to various fields of automobile manufacturing, 3C electronics, food processing and the like. However, automated grasping of flexible target objects (e.g., wire harnesses, flexible circuit boards, cables, etc.) still face serious technical challenges. The flexible target object is extremely easy to be influenced by various external disturbance in the grabbing process due to the characteristics of low rigidity and easy deformation of the flexible target object, and the random disturbance comprises inertia force disturbance, environment foundation vibration, airflow impact and the like generated by the motion of the robot. These multi-source perturbations have complex spatio-temporal correlations that together result in nonlinear, unpredictable displacement shifts at the end regions of the target object, severely affecting the success rate and accuracy of the capture. In the prior art, the traditional visual servo method generally adopts image-based characteristic feedback to carry out closed-loop control, but the method has lag response to high-frequency and small-amplitude nonlinear motion, and is difficult to realize real-time compensation. Although the control performance is improved to a certain extent by the model-based predictive control method, the adopted linear model is difficult to accurately describe the nonlinear dynamic behavior of the flexible body, and decoupling and targeted processing on multi-source disturbance are lacking. In addition, the prior art often ignores stability constraints during the grabbing process, resulting in the risk of grabbing slip or target damage during the dynamic compensation process. Disclosure of Invention The invention aims to provide a robot grabbing system and a robot grabbing method based on computer vision, which are used for solving the technical problems that in the prior art, nonlinear offset of a flexible target object in complex space-time correlation is difficult to accurately identify and compensate, and grabbing stability is difficult to ensure in a dynamic compensation process. To achieve the above object, according to one aspect of the present invention, there is provided a robot gripping method based on computer vision, comprising the steps of: Acquiring a high-speed visual image sequence of the tail end part of the flexible target object acquired by a visual sensor in real time, and acquiring multi-source disturbance data reflecting the environment and the state of the robot system; extracting space-time characteristics of the high-speed visual image sequence to obtain the space-time characteristics of the tail end displacement of the tail end part in a preset history window before the current moment; based on the multisource disturbance data and the terminal displacement space-time characteristics, constructing a nonlinear dynamic model describing dynamic behaviors of the flexible target object under the action of external excitation; Taking the nonlinear dynamic model as a prediction model, combining the actual displacement state of the tail end part at the current moment, and calculating to obtain an optimal control compensation sequence for compensating nonlinear displacement offset caused by multi-source disturbance by solving a constraint optimization problem in a finite time domain; Extracting control compensation at the current moment from the optimal control compensation sequence, generating a control instruction and sending the control instruction to a motion controller of the robot so as to drive an end effector of the robot to adjust a grabbing track; And repeatedly executing the steps at the next moment to realize the rolling optimization control. Further, the multi-source disturbance data comprise robot joint moment data acquired by a robot joint moment sensor and environmental vibration data of a grabbing station acquired by an environmental vibration sensor; The robot grabbing method based on computer vision further comprises the following steps: Performing spectrum analysis on the robot joint moment data to obtain a deterministic disturbance sequence caused by robot motion; Performing time-frequency analysis on the environmental vibration data to obtain a periodic disturbance sequence caused by environmental vibration; the deterministic perturbation sequence and the periodic perturbation sequence are used as external excitation input of the nonlinear dynamic model. Further, extracting space-time characteristics of the high-speed visual image sequence to obtain the space-time characteristics of the tail end displacement of the tail end part in a preset history wi