CN-121999481-A - Fruit maturity detection and grading picking method and system

CN121999481ACN 121999481 ACN121999481 ACN 121999481ACN-121999481-A

Abstract

The invention provides a fruit maturity detection and grading picking method and system, which comprise the steps of adopting a network of parallel color texture and spatial contour attention modules to extract a fruit image fusion feature map, enhancing color texture and boundary information, generating an initial mask based on the feature map, establishing a joint prediction model for a mask overlapping region, calculating membership probability of overlapping pixels to adjacent examples by utilizing feature map boundary response and non-overlapping region contour curvature, distributing pixels to obtain complete shape masks of all fruits, constructing a fruit maturity feature vector based on the complete masks, evaluating shielding degree, determining the shielding degree by the area proportion of the overlapping region of the non-overlapping region and the home fruits, inputting the feature vector into a classifier to judge maturity grade, and establishing a cost function for up-to-standard fruits to generate an optimal picking sequence.

Inventors

Zhang Danran
ZHENG ZHENHUI
WEI LIJIAO
WANG SHUO
HUANG WEIHUA
DU DONGJIE
LI MING

Assignees

华南农业大学
中国热带农业科学院农业机械研究所

Dates

Publication Date: 20260508
Application Date: 20260206

Claims (10)

1. The fruit maturity detection and grading picking method is characterized by comprising the following steps of: Extracting a fused attention feature map of a fruit image by adopting a feature extraction network of which a color texture attention module and a spatial contour attention module are arranged in parallel, and enhancing fruit color textures and spatial boundary information in the feature map; generating an initial mask of each fruit based on the fused attention feature map, establishing a joint prediction model for an overlapping area between the masks, calculating the membership probability of each pixel belonging to adjacent examples in the overlapping area, and distributing the pixels to the example with the highest membership probability to generate a complete shape mask of each fruit; The method comprises the steps of generating a first characteristic sub-vector on a color texture detail characteristic map through pooling operation on a non-overlapping region in a fruit complete shape mask, aggregating color texture characteristics according to membership probability corresponding to each pixel point in the overlapping region in the fruit complete shape mask to generate a second characteristic sub-vector, splicing the first characteristic sub-vector and the second characteristic sub-vector to obtain a maturity characteristic vector, and dividing the number of pixels in the overlapping region belonging to the fruit by the total number of pixels in the complete shape mask to obtain a value serving as shielding degree; and for the fruits reaching the picking standard, establishing a cost function at least according to the shielding degree and the maturity grade to generate an optimal picking sequence.
2. The method of claim 1, wherein the extracting the fused attention profile of the fruit image using the feature extraction network of the parallel arrangement of the color texture attention module and the spatial contour attention module comprises: the color texture attention module is composed of a plurality of convolution layers and is used for extracting color and texture detail feature images of fruits; the spatial contour attention module strengthens a boundary contour feature map of the fruit through an edge detection operator and pooling operation; And splicing and fusing the color texture detail feature map and the boundary contour feature map, and outputting the fused attention feature map.
3. The method of claim 1, wherein the generating an initial mask for each fruit based on the fused attention profile comprises: Performing multi-scale feature mapping on the fused attention feature map, and extracting local and whole response features of fruits by setting convolution kernels of different receptive fields; based on the multi-scale feature mapping, a pixel-level instance segmentation prediction network is adopted to judge the foreground and the background of each pixel in the fused attention feature map, so as to obtain a probability response map of a fruit candidate region; Threshold segmentation and connected domain analysis are carried out on the probability response graph, and candidate areas corresponding to each fruit instance are generated; and executing morphological operation on the candidate region to eliminate noise and fill the hole, so as to obtain an initial mask of each fruit.
4. The method according to claim 1 or 2, wherein the establishing a joint prediction model for the overlapping region between the masks, calculating the membership probability of each pixel in the overlapping region to a neighboring instance, includes: Calculating boundary response values of pixels in the overlapped area in the fused attention feature map; Extracting the outline of the adjacent examples in the non-overlapping area, and calculating an outline curvature index; and determining the membership probability according to the boundary response value, the outline curvature index of the belonging example and the distance from the pixel point to the outline of the non-overlapping area of the example, and carrying out normalization processing.
5. The method of claim 1, wherein the aggregating the color texture features according to the membership probabilities corresponding to the pixels for the overlapping regions in the fruit full shape mask to generate a second feature sub-vector comprises: for each pixel point in the overlapping area, acquiring a corresponding pixel level feature of the pixel point on the color texture detail feature map; Multiplying the pixel-level feature by the membership probability of the pixel point belonging to the current target fruit instance to obtain a weighted feature of the pixel point; and carrying out summation or pooling operation on all weighted features belonging to the pixel points in the overlapping area of the current target fruit instance, thereby obtaining the second feature sub-vector.
6. The method of claim 1, wherein said inputting the maturity feature vector into a maturity classifier determines a maturity rating of each fruit, comprising: The maturity classifier is a support vector machine classifier; A plurality of maturity levels are predefined, and the classifier maps the input maturity feature vector to one of the levels.
7. A fruit maturity detection and grading picking system comprising the following modules: the enhancement module is used for extracting a fusion attention feature map of the fruit image by adopting a feature extraction network of the parallel color texture attention module and the spatial contour attention module, and enhancing fruit color textures and spatial boundary information in the feature map; The distribution module is used for generating an initial mask of each fruit based on the fused attention feature map, establishing a joint prediction model for an overlapping area between the masks, calculating the membership probability of each pixel in the overlapping area to an adjacent instance, and distributing the pixel to the instance with the highest membership probability to generate a complete shape mask of each fruit; The construction module is used for generating a first characteristic sub-vector on the color texture detail characteristic map through pooling operation on a non-overlapping region in the fruit complete shape mask, aggregating color texture characteristics according to membership probability corresponding to each pixel point on the overlapping region in the fruit complete shape mask to generate a second characteristic sub-vector, splicing the first characteristic sub-vector and the second characteristic sub-vector to obtain a maturity characteristic vector, and dividing the number of pixels of the overlapping region belonging to the fruit by the total number of pixels of the complete shape mask to obtain a value serving as shielding degree; And for the fruits reaching the picking standard, establishing a cost function according to at least the shielding degree and the maturity grade to generate an optimal picking sequence.
8. The system of claim 7, wherein the feature extraction network employing the parallel arrangement of the color texture attention module and the spatial contour attention module extracts a fused attention feature map of the fruit image, comprising: the color texture attention module is composed of a plurality of convolution layers and is used for extracting color and texture detail feature images of fruits; the spatial contour attention module strengthens a boundary contour feature map of the fruit through an edge detection operator and pooling operation; And splicing and fusing the color texture detail feature map and the boundary contour feature map, and outputting the fused attention feature map.
9. The system of claim 7, wherein the generating an initial mask for each fruit based on the fused attention profile comprises: Performing multi-scale feature mapping on the fused attention feature map, and extracting local and whole response features of fruits by setting convolution kernels of different receptive fields; based on the multi-scale feature mapping, a pixel-level instance segmentation prediction network is adopted to judge the foreground and the background of each pixel in the fused attention feature map, so as to obtain a probability response map of a fruit candidate region; Threshold segmentation and connected domain analysis are carried out on the probability response graph, and candidate areas corresponding to each fruit instance are generated; and executing morphological operation on the candidate region to eliminate noise and fill the hole, so as to obtain an initial mask of each fruit.
10. The system of claim 7 or 8, wherein the establishing a joint prediction model for the overlapping region between the masks, calculating the membership probability of each pixel in the overlapping region to a neighboring instance, comprises: Calculating boundary response values of pixels in the overlapped area in the fused attention feature map; Extracting the outline of the adjacent examples in the non-overlapping area, and calculating an outline curvature index; and determining the membership probability according to the boundary response value, the outline curvature index of the belonging example and the distance from the pixel point to the outline of the non-overlapping area of the example, and carrying out normalization processing.

Description

Fruit maturity detection and grading picking method and system Technical Field The application belongs to the field of vision, and particularly relates to a fruit maturity detection and grading picking method and system. Background Fruit maturity detection and grading picking technology based on machine vision generally adopts image processing and deep learning methods to identify and analyze fruits in an orchard environment. However, when fruits are blocked from each other, it is difficult for an image segmentation algorithm or a general example segmentation network to separate each fruit, resulting in blocking of segmentation masks, edge defects, or loss of objects. Because all visual information of the fruits cannot be obtained, when a key area for judging the maturity is shielded, misjudgment on the fruit maturity is easily caused. In addition, most picking systems, after determining the picking targets, generate picking sequences based on the determined maturity levels without incorporating positional information of the robotic arm. Meanwhile, the operability of the fruits, such as whether the shape and the gesture are favorable for the grabbing factors of the manipulator, is also rarely analyzed. The single-dimension decision mode leads to that the planned picking path is not optimal when the picking robot faces dense fruit clusters, the operation efficiency is low, even the picking robot is easy to collide with surrounding branches and leaves or other fruits when picking, the picking success rate is reduced, and the fruits and plants are possibly damaged. Therefore, how to fuse multidimensional information to make intelligent picking decisions under the condition of serious shielding is a technical problem to be solved in the current automatic picking field. Disclosure of Invention The invention provides a fruit maturity detection and grading picking method, which is used for solving the problem that the prior art cannot carry out intelligent picking decision by fusing multidimensional information under the condition of serious shielding among fruits, and comprises the following steps: Extracting a fused attention feature map of a fruit image by adopting a feature extraction network of which a color texture attention module and a spatial contour attention module are arranged in parallel, and enhancing fruit color textures and spatial boundary information in the feature map; generating an initial mask of each fruit based on the fused attention feature map, establishing a joint prediction model for an overlapping area between the masks, calculating the membership probability of each pixel belonging to adjacent examples in the overlapping area, and distributing the pixels to the example with the highest membership probability to generate a complete shape mask of each fruit; The method comprises the steps of generating a first characteristic sub-vector on a color texture detail characteristic map through pooling operation on a non-overlapping region in a fruit complete shape mask, aggregating color texture characteristics according to membership probability corresponding to each pixel point in the overlapping region in the fruit complete shape mask to generate a second characteristic sub-vector, splicing the first characteristic sub-vector and the second characteristic sub-vector to obtain a maturity characteristic vector, and dividing the number of pixels in the overlapping region belonging to the fruit by the total number of pixels in the complete shape mask to obtain a value serving as shielding degree; and for the fruits reaching the picking standard, establishing a cost function at least according to the shielding degree and the maturity grade to generate an optimal picking sequence. In addition, the invention also relates to a fruit maturity detection and grading picking system, which comprises the following modules: the enhancement module is used for extracting a fusion attention feature map of the fruit image by adopting a feature extraction network of the parallel color texture attention module and the spatial contour attention module, and enhancing fruit color textures and spatial boundary information in the feature map; The distribution module is used for generating an initial mask of each fruit based on the fused attention feature map, establishing a joint prediction model for an overlapping area between the masks, calculating the membership probability of each pixel in the overlapping area to an adjacent instance, and distributing the pixel to the instance with the highest membership probability to generate a complete shape mask of each fruit; The construction module is used for generating a first characteristic sub-vector on the color texture detail characteristic map through pooling operation on a non-overlapping region in the fruit complete shape mask, aggregating color texture characteristics according to membership probability corresponding to each pixel point on the overlapping region