CN-118809608-B - Object sorting method and arm-hand system based on multimode perception and hardness touch detection

CN118809608BCN 118809608 BCN118809608 BCN 118809608BCN-118809608-B

Abstract

An object sorting method based on multimode perception and hardness touch detection and an arm-hand system, wherein the method comprises the steps of 1, constructing a training set of a fruit hardness classification network, training the fruit hardness classification network, 2, calibrating a depth camera by hand and eye, mapping a target position visually perceived by the depth camera onto a motion track of a mechanical arm, 3, training a YOLOv model, 4, grabbing target fruits by the mechanical arm under the assistance of the depth camera and the trained YOLOv model, transmitting touch information of a first flexible touch sensor to the trained fruit hardness classification network to obtain the hardness of the target fruits, and 5, classifying the maturity of the fruits and sorting the fruits to a designated position. The invention utilizes the vision guiding arm hand system to grasp, realizes the full-touch maturity detection of the target fruit, improves the detection efficiency, and has good universality in life and production scenes.

Inventors

CHEN WENRUI
ZHOU ZELONG
WANG YAONAN
DIAO QIANG
HOU LIWEI

Assignees

湖南大学

Dates

Publication Date: 20260505
Application Date: 20240815

Claims (8)

1. An object sorting method based on multimode sensing and hardness touch detection is characterized by comprising the following steps: S1, firstly, constructing a training set of a fruit hardness classification network by utilizing a fruit maturity detection module, and training the fruit hardness classification network by utilizing the training set to obtain a trained fruit hardness classification network; s2, performing hand-eye calibration on the depth camera, and mapping the target position visually perceived by the depth camera onto the motion track of the mechanical arm; S3, training the YOLOv model in the upper computer to obtain a trained YOLOv model; s4, under the joint assistance of the depth camera and the trained YOLOv model, the mechanical arm grabs the target fruit by using a mechanical clamping jaw at the tail end of the mechanical arm, a first flexible touch sensor on the mechanical clamping jaw senses the hardness information of the target fruit, and the obtained touch information is transmitted to a trained fruit hardness classification network, so that the hardness of the target fruit is obtained; S5, classifying the target fruit maturity according to the fruit types obtained through recognition of the trained YOLOv model and the hardness level dividing line of the corresponding fruit, and sorting to a designated position through a mechanical arm; the training set for constructing the fruit hardness classification network by utilizing the fruit maturity detection module in the S1 specifically comprises the following steps: S11, randomly selecting a plurality of detection points in a plane area at the top of the double-shaft movable platform, wherein a detected object with a corresponding hardness level is placed and clamped at the top of the double-shaft movable platform in the fruit maturity detection module; S12, rotating a hand wheel on the side surface of the testing machine frame to enable the movable end of the testing machine frame to drive the push-pull force meter to move vertically until the detection end of the push-pull force meter contacts with a detected object and the pressure applied by the push-pull force meter reaches a set value so as to simulate an initial gripping state; s13, adjusting the positions of the detected object and the detection end of the push-pull force meter through the double-shaft moving platform until the detection end of the push-pull force meter is positioned right above another detection point, and continuously driving the push-pull force meter to move downwards for a fixed stroke after the adjustment is completed to obtain pressure data of the second flexible touch sensor positioned at the detection point; s14, circulating S13 until pressure data of all detection points in the plane area are obtained, so that a training set of the fruit hardness classification network is obtained; the step S2 specifically comprises the following steps: S21, fixing the depth camera at a position outside the mechanical arm and keeping the depth camera still, and keeping the relative position between the calibration pattern and the tail end of the mechanical arm unchanged; S22, moving the mechanical arm, photographing the calibration pattern for a plurality of times by using the depth camera, and recording the homogeneous transformation matrix of the calibration pattern relative to the depth camera every time And homogeneous transformation matrix of the manipulator end effector relative to the manipulator base ; S23, then according to the calibration image, the homogeneous transformation matrix of the manipulator base is opposite to that of the manipulator base The transformation matrix of the depth camera relative to the mechanical arm substrate is unchanged and solved by the following equation Further completing the hand-eye calibration process of the depth camera, and mapping the target position visually perceived by the depth camera onto the motion trail of the mechanical arm; Wherein, the 、 And Representing homogeneous transformation matrix of the mechanical arm end effector relative to the mechanical arm base at three different positions after the mechanical arm is moved, 、 And And after the mechanical arm is moved, the calibration patterns at three different positions are represented by homogeneous transformation matrixes of the depth camera.
2. The object sorting method according to claim 1, wherein S4 specifically comprises the steps of: s41, transmitting visual information to a trained YOLOv model in an upper computer control unit by a depth camera, and confirming grabbing points of target fruits by a posture estimator in the trained YOLOv model according to the visual information transmitted by the depth camera; s42, the upper computer control unit generates control information according to the grabbing points of the target fruits and transmits the control information to the mechanical arm, and the mechanical arm grabs the target fruits by using mechanical clamping jaws at the tail end of the mechanical arm; s43, sensing the hardness information of the target fruit by a first flexible touch sensor on the mechanical clamping jaw, and transmitting the obtained touch information to a trained fruit hardness classification network to further obtain the hardness of the target fruit.
3. The object sorting method according to claim 1, further comprising the steps of: S6, circulating the steps S4 to S5 until the maturity information of all the fruits is obtained, and sorting all the fruits to the designated positions according to the maturity information of all the fruits.
4. An arm and hand system for sorting by the object sorting method according to any one of claims 1 to 3, comprising in particular: the mechanical arm is arranged in the object sorting area through a mechanical arm base; the mechanical clamping jaw is arranged at the tail end of the mechanical arm and is used for grabbing fruits; the first flexible touch sensor is fixedly arranged at the fingertip part of the mechanical clamping jaw through the base; the depth camera is erected on the periphery of the mechanical arm to acquire visual information of fruits in real time; The upper computer control unit is respectively and electrically connected with the mechanical arm, the mechanical clamping jaw, the first flexible touch sensor and the depth camera so as to control the mechanical arm and the mechanical clamping jaw, and the upper computer control unit is internally provided with a fruit hardness classification network and a YOLOv model.
5. The arm hand system of claim 4, wherein the robotic arm is a six degree of freedom robotic arm.
6. The arm hand system of claim 4, wherein the mechanical clamping jaw is an articulated self-adaptive electric jaw, and the articulated self-adaptive electric jaw carries a force control and programmable control unit.
7. The arm hand system of claim 4, wherein the first flexible tactile sensor is a flexible fingertip tactile sensor based on a4 x 4 array of barometer sensors.
8. The arm hand system of any one of claims 4 to 7, wherein the fruit hardness classification network comprises a 1-D convolution layer with ReLU activation, dropou layers, and an average pooling layer connected in sequence.

Description

Object sorting method and arm-hand system based on multimode perception and hardness touch detection Technical Field The invention relates to the technical field of object sorting, in particular to an object sorting method and an arm-hand system based on multimode perception and hardness touch detection. Background Hardness is a key indicator in evaluating material quality and manufacturing processes. Accurate hardness identification can help manufacturers ensure consistency and reliability of products, thereby improving product quality and market competitiveness. The hardness data may provide valuable information about the behavior and properties of the material, such as its wear resistance, fatigue resistance, and deformation characteristics. Such information is of great importance in predicting the performance of materials under different environmental conditions. Fruit firmness recognition is an important task in the agricultural and food industries, mainly for assessing the ripeness, quality and eating state of fruits. The Chinese is a large country for producing fruits and vegetables, and plays an important role in national economy. However, about three fruits are wasted every year for various reasons. Wherein waste during picking, storage and transportation occupies a large proportion. At present, the main fruit maturity detection technology mainly relies on fruit farmers to judge empirically or detect manually by means of a sugar degree acidity analyzer, a hardness meter and other tools, and judges the maturity grade of the fruit according to the relevant standard. While the destructively detected fruit cannot be eaten or sold. According to the process, the fruit maturity extraction and detection work has high labor cost, low efficiency, subjectivity and instability in detection, and cannot be performed in large-scale real-time detection. The current work for predicting and classifying fruit maturity aims to address these issues, such as some machine learning based approaches. But these methods often require specific environments, complex equipment, extensive data processing, etc. There is still a gap between the realization of true intelligent automatic detection and maturity classification. The sensor can be used for interaction and detection of an object to be detected. However, the development of the current flexible sensor has the limitations of area, sensing range and the like. There are certain limitations on the type, size and weight of fruit that can be measured, particularly the planar fingertip flexible tactile sensor, which makes it difficult to achieve adequate contact with specially shaped fruits. Disclosure of Invention The invention provides an object sorting method and an arm-hand system based on multimode sensing and hardness touch detection, which are used for solving the technical problems mentioned in the background art. In order to achieve the above purpose, the technical scheme of the invention is realized as follows: The invention provides an object sorting method based on multimode perception and hardness touch detection, which comprises the following steps: S1, firstly, constructing a training set of a fruit hardness classification network by utilizing a fruit maturity detection module, and training the fruit hardness classification network by utilizing the training set to obtain a trained fruit hardness classification network; s2, performing hand-eye calibration on the depth camera, and mapping the target position visually perceived by the depth camera onto the motion track of the mechanical arm; S3, training the YOLOv model in the upper computer to obtain a trained YOLOv model; s4, under the joint assistance of the depth camera and the trained YOLOv model, the mechanical arm grabs the target fruit by using a mechanical clamping jaw at the tail end of the mechanical arm, a first flexible touch sensor on the mechanical clamping jaw senses the hardness information of the target fruit, and the obtained touch information is transmitted to a trained fruit hardness classification network, so that the hardness of the target fruit is obtained; s5, classifying the target fruit maturity according to the fruit types obtained through recognition of the trained YOLOv model and the hardness level dividing line of the corresponding fruit, and sorting to a designated position through a mechanical arm. Further, the training set for constructing the fruit hardness classification network by using the fruit maturity detection module in S1 specifically includes the following steps: S11, randomly selecting a plurality of detection points in a plane area at the top of the double-shaft movable platform, wherein a detected object with a corresponding hardness level is placed and clamped at the top of the double-shaft movable platform in the fruit maturity detection module; S12, rotating a hand wheel on the side surface of the testing machine frame to enable the movable end of the testing machine f