CN-121989247-A - Intelligent robot grabbing method and device, electronic equipment and storage medium

CN121989247ACN 121989247 ACN121989247 ACN 121989247ACN-121989247-A

Abstract

The application relates to an intelligent robot grabbing method, an intelligent robot grabbing device, electronic equipment and a storage medium. The method comprises the steps of obtaining multi-mode sensing information corresponding to a target grabbing scene, conducting semantic analysis processing on the multi-mode sensing information based on a multi-mode semantic analysis model, outputting semantic association information corresponding to an object to be grabbed, determining a target grabbing object, recommended grabbing points corresponding to the target grabbing object and a recommended grabbing strategy according to the semantic association information, conducting pose prediction processing on the recommended grabbing points based on a grabbing pose prediction network to obtain robot grabbing pose information, conducting grabbing processing on the target grabbing object according to the recommended grabbing strategy and the robot grabbing pose information to obtain grabbing state parameters, and conducting adjustment processing on the robot grabbing pose information based on the grabbing state parameters. According to the technical scheme provided by the application, the object can be effectively prevented from sliding, damaging or colliding, and the intelligent level and the grabbing efficiency of the robot are improved.

Inventors

HE YANG
YU GUITAO

Assignees

宁波方太厨具有限公司

Dates

Publication Date: 20260508
Application Date: 20260310

Claims (10)

1. An intelligent robot grabbing method is characterized by comprising the following steps: Acquiring multi-mode sensing information corresponding to a target grabbing scene; based on a multi-mode semantic analysis model, carrying out semantic analysis processing on the multi-mode perception information, and outputting semantic association information corresponding to the object to be grabbed; determining a target grabbing object, recommended grabbing points corresponding to the target grabbing object and a recommended grabbing strategy according to the semantic association information; Based on a grabbing pose prediction network, carrying out pose prediction processing on the recommended grabbing points to obtain grabbing pose information of the robot; according to the recommended grabbing strategy and the robot grabbing pose information, grabbing the target grabbing object to obtain grabbing state parameters; and based on the grabbing state parameters, adjusting the grabbing pose information of the robot.
2. The intelligent robot gripping method according to claim 1, wherein the semantic association information includes an object stacking hierarchy, an object material property, an object geometry, object space orientation information, and an object abnormal state detection result, and the determining, according to the semantic association information, a target gripping object, a recommended gripping point corresponding to the target gripping object, and a recommended gripping policy includes: According to the object stacking level, determining the object with the highest grabbing priority as the target grabbing object; determining a candidate grabbing area on the target grabbing object based on the object material attribute, the object geometric shape and the object space orientation information; Removing an oil pollution abnormal region and a damaged abnormal region from the candidate grabbing region according to the object abnormal state detection result to obtain the recommended grabbing point; And determining the recommended grabbing strategy according to the position information of the recommended grabbing point.
3. The intelligent robot gripping method according to claim 1, wherein the semantic association information includes object stacking layers, object material properties, object geometry, object space orientation information, and object abnormal state detection results, the semantic analysis processing is performed on the multimodal perception information based on the multimodal semantic analysis model, and semantic association information corresponding to an object to be gripped is output, including: Inputting a target color image sequence in the multi-mode sensing information into a target detection network in the multi-mode semantic analysis model to perform target recognition processing, so as to obtain the types of the objects to be captured and the position information of the objects to be captured in the target capturing scene; Inputting a target depth image sequence in the multi-mode sensing information and the position information of each object to be grabbed into a space analysis network in the multi-mode semantic analysis model to perform space relation processing, so as to obtain the object stacking level between each object to be grabbed; inputting the target color image sequence into a material identification network in the multi-mode semantic analysis model to perform surface attribute processing to obtain the object material attribute of each object to be grabbed; Inputting the target depth image sequence into a geometric reconstruction network in the multi-mode semantic analysis model to perform three-dimensional shape processing to obtain the object geometric shape and the object space orientation information; Inputting the target color image sequence to an anomaly detection network in the multi-mode semantic analysis model for surface anomaly processing to obtain object anomaly detection results of the objects to be grabbed.
4. The intelligent robot gripping method according to claim 1, wherein the performing, based on the gripping pose prediction network, pose prediction processing on the recommended gripping point to obtain the robot gripping pose information includes: Extracting local point cloud data corresponding to the recommended grabbing points according to a point cloud extractor in the grabbing pose prediction network; Encoding the local point cloud data according to a grabbing sampler in the grabbing pose prediction network to generate a plurality of candidate grabbing poses; And carrying out collision evaluation on the plurality of candidate grabbing postures based on a grabbing evaluator in the grabbing posture prediction network, and screening out the robot grabbing posture information from the plurality of candidate grabbing postures.
5. The method for capturing the intelligent robot according to claim 1, wherein the capturing state parameters include capturing mechanical parameters and capturing visual parameters, and the adjusting the capturing pose information of the robot based on the capturing state parameters comprises: Under the condition that the grabbing mechanical parameter exceeds the mechanical threshold range, adjusting the grabbing force of the robot in the grabbing pose information of the robot; and under the condition that the grabbing visual parameters exceed the visual deviation threshold, adjusting the grabbing position orientation of the robot in the grabbing pose information of the robot.
6. The intelligent robotic grasping method of claim 5, wherein the method further comprises: Generating a target robot grabbing instruction according to the adjusted robot grabbing force and the adjusted robot grabbing position orientation; And responding to the grabbing instruction of the target robot, and controlling the robot to grab the target grabbing object again.
7. The method for capturing an intelligent robot according to claim 1, wherein the acquiring the multi-modal sensing information corresponding to the target capturing scene includes: Acquiring an initial color image sequence corresponding to the target grabbing scene and an initial depth image sequence corresponding to the target grabbing scene in multiple directions; Under the condition that the image quality of the initial color image sequence does not meet the preset image quality threshold, re-acquisition processing is carried out on the color images in the corresponding directions in the initial color image sequence to obtain a target color image sequence; Under the condition that the image quality of the initial depth image sequence does not meet the preset image quality threshold, re-acquiring depth images in corresponding directions in the initial depth image sequence to obtain a target depth image sequence; And denoising and space-time registration processing are carried out on the target color image sequence and the target depth image sequence, so as to generate multi-mode perception information.
8. An intelligent robotic grasping device, comprising: the information acquisition module is used for acquiring multi-mode sensing information corresponding to the target grabbing scene; The semantic association information determining module is used for carrying out semantic analysis processing on the multi-mode sensing information based on the multi-mode semantic analysis model and outputting semantic association information corresponding to the object to be grabbed; the recommendation grabbing strategy determining module is used for determining a target grabbing object, a recommendation grabbing point corresponding to the target grabbing object and a recommendation grabbing strategy according to the semantic association information; The grabbing pose information determining module is used for carrying out pose prediction processing on the recommended grabbing points based on a grabbing pose prediction network to obtain grabbing pose information of the robot; the grabbing state determining module is used for carrying out grabbing processing on the target grabbing object according to the recommended grabbing strategy and the robot grabbing pose information to obtain grabbing state parameters; and the grabbing pose adjusting module is used for adjusting the grabbing pose information of the robot based on the grabbing state parameters.
9. An electronic device, comprising: A processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the intelligent robot gripping method of any of claims 1 to 7.
10. A computer readable storage medium, characterized in that instructions in the computer readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the intelligent robot gripping method according to any of claims 1 to 7.

Description

Intelligent robot grabbing method and device, electronic equipment and storage medium Technical Field The present application relates to the field of artificial intelligence technologies, and in particular, to an intelligent robot grabbing method, an intelligent robot grabbing device, an electronic device, and a storage medium. Background With the rapid development of artificial intelligence and robot technology, intelligent robots are increasingly used in the fields of industrial sorting, logistics storage, home service, dangerous environment operation and the like. The robot grabbing technology is used as a core link for physical interaction between the robot and the environment, and the success rate and the adaptability of the robot directly determine the operation efficiency and the application range of the robot. Traditional robotic grasping methods rely primarily on preprogrammed fixed trajectories or recognition algorithms based on simple geometric features, such as shape, size. Such algorithms generally perform well in structured environments, but when facing complex scenes with unstructured, dynamic changes or various objects, the algorithms tend to increase pose estimation errors, have the problems of low grabbing accuracy, poor adaptability and insufficient efficiency, and are difficult to meet the increasingly complex actual application demands. Disclosure of Invention The application provides an intelligent robot grabbing method, an intelligent robot grabbing device, electronic equipment and a storage medium, which at least solve the problems of how to improve the accuracy and the intelligent level of robot grabbing in the related technology. The technical scheme of the application is as follows: According to a first aspect of an embodiment of the present application, there is provided an intelligent robot gripping method, including: Acquiring multi-mode sensing information corresponding to a target grabbing scene; based on a multi-mode semantic analysis model, carrying out semantic analysis processing on the multi-mode perception information, and outputting semantic association information corresponding to the object to be grabbed; determining a target grabbing object, recommended grabbing points corresponding to the target grabbing object and a recommended grabbing strategy according to the semantic association information; Based on a grabbing pose prediction network, carrying out pose prediction processing on the recommended grabbing points to obtain grabbing pose information of the robot; according to the recommended grabbing strategy and the robot grabbing pose information, grabbing the target grabbing object to obtain grabbing state parameters; and based on the grabbing state parameters, adjusting the grabbing pose information of the robot. According to a second aspect of an embodiment of the present application, there is provided an intelligent robot gripping apparatus including: the information acquisition module is used for acquiring multi-mode sensing information corresponding to the target grabbing scene; The semantic association information determining module is used for carrying out semantic analysis processing on the multi-mode sensing information based on the multi-mode semantic analysis model and outputting semantic association information corresponding to the object to be grabbed; the recommendation grabbing strategy determining module is used for determining a target grabbing object, a recommendation grabbing point corresponding to the target grabbing object and a recommendation grabbing strategy according to the semantic association information; The grabbing pose information determining module is used for carrying out pose prediction processing on the recommended grabbing points based on a grabbing pose prediction network to obtain grabbing pose information of the robot; the grabbing state determining module is used for carrying out grabbing processing on the target grabbing object according to the recommended grabbing strategy and the robot grabbing pose information to obtain grabbing state parameters; and the grabbing pose adjusting module is used for adjusting the grabbing pose information of the robot based on the grabbing state parameters. According to a third aspect of embodiments of the present application there is provided an electronic device comprising a processor, a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement a method as in any of the first aspects above. According to a fourth aspect of embodiments of the present application, there is provided a computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the method of any of the first aspects of embodiments of the present application. According to a fifth aspect of embodiments of the present application, there is provided a computer program product c