CN-122023934-A - Hierarchical association tracking and defect quantifying method and system for rotating spherical fruit target

Abstract

The invention discloses a hierarchical association tracking and defect quantifying method and a hierarchical association tracking and defect quantifying system for a rotary spherical fruit target. The method comprises the steps of collecting conveyor belt videos, detecting and dividing to obtain fruit boundary frames and defect masks, establishing TID and QTID levels according to the containing relation, adopting Ma distance motion gating and EMA updating ReID cosine gating, carrying out multi-frame confirmation, weighting the existence degree threshold by confidence, eliminating false detection, merging TIDs, distributing unique logic IDs, mapping the defect masks to fruit local coordinates, rasterizing, de-duplicating the same physical defects based on signature IoU threshold, determining representation QTID according to the historical maximum area, establishing alias mapping, de-duplicating the defect union during track merging, determining and grading the defect areas represented QTID under statistical logic IDs, and outputting the result to a PLC or an executing mechanism. The system comprises image acquisition, detection segmentation, hierarchical binding, track association, defect de-duplication, hierarchical decision and control output units, and improves track stability and defect statistical consistency.

Inventors

• YUAN TING
• WANG SHITING
• XIONG ZIMING
• Yuan Jiace
• Ai Xinyi
• RONG JIACHENG
• HOU BOYUAN
• ZHANG JUNXIONG
• ZHANG WENQIANG

Assignees

• 中国农业大学

Dates

Publication Date

20260512

Application Date

20260214

Claims (14)

1. 1. The method for grading association tracking and defect quantification of the rotating spherical fruit target is characterized by comprising the following steps of: Acquiring a video sequence of a detection area of the roller conveyor belt; Performing target detection and example segmentation on each frame of image of the video sequence to obtain a boundary frame of a parent fruit target and a pixel mask of a child defect target, and obtaining respective confidence scores; Binding the defect mask to a fruit target to which the defect mask belongs according to the spatial inclusion relation between the defect mask and the fruit target boundary box, and establishing a father-son hierarchy data structure of a fruit track mark TID and a defect track mark QTID; Track association is carried out on fruit targets, a Kalman filtering prediction model is established for each fruit track, the square Markov distance between current observation and prediction is calculated, and the square Markov distance and the motion consistency threshold are gated Comparing to screen candidate trajectories; Extracting feature vectors of fruit appearance ReID, updating appearance prototype vectors by adopting exponential moving average, calculating cosine distance between appearance prototype vectors and gating threshold value consistent with appearance Comparing to screen candidate trajectories; Carrying out multi-frame confirmation on candidate track pairs which are simultaneously subjected to motion consistency gating and appearance consistency gating, weighting the confidence coefficient of a plurality of TIDs belonging to the same physical fruit instance to the sum S of the existence degrees after confirmation, confirming that the fruit instance is an effective target when the S exceeds a preset threshold value, and distributing a unique logic ID; performing topology deduplication on a defect target by mapping a defect mask bound to the same fruit target to a local coordinate system of the fruit target and rasterizing the defect mask into a binary feature grid with a fixed size of N×N; calculating a signature cross-over ratio for the candidate defects, and when the signature cross-over ratio is not smaller than a signature threshold value When the defect observation under different visual angles is related to the same physical defect; Selecting QTID with the largest historical defect area as a representation QTID of the physical defect for a defect observation set judged to be the same physical defect, and establishing alias mapping to enable other QTID in the set to point to the representation QTID; Performing union deduplication on the defect list based on the delegate QTID and attributing the delegate QTID to the logical ID when TID merging occurs; And counting the accumulated defect areas of all representatives QTID under the logic ID, comparing the accumulated defect areas with a preset grading threshold to obtain grading results, generating a control instruction by the grading results, and sending the control instruction to a PLC or an executing mechanism to finish sorting execution.
2. 2. The method of claim 1, wherein binding the defect mask to its belonging fruit target according to its spatial inclusion relation with the fruit target bounding box comprises: And binding the defect mask to a certain fruit target when the geometric center point is positioned inside a boundary box of the fruit target.
3. 3. The method of claim 1, wherein performing track association on fruit targets further comprises time gating, the method further comprising: Selecting a candidate track from a history track buffer pool and calculating the frame interval between the current track and the candidate track Only when And when the candidate track is not larger than the preset time gating threshold value, carrying out Kalman prediction on the candidate track and entering into associated screening.
4. 4. The method of claim 1, wherein the state vector of the kalman filter predictive model is an 8-dimensional vector including bounding box center coordinates, aspect ratio, altitude and rate of change thereof, the motion consistency gating threshold value For gating the squared mahalanobis distance.
5. 5. The method of claim 1, wherein updating the appearance prototype vector using exponential sliding average comprises, when the trajectory obtains the appearance ReID feature vector at the current frame, following Is updated by means of the (a) and (b), And when the track does not obtain the effective appearance ReID characteristic vector in the current frame, freezing the appearance prototype vector of the previous frame and not updating.
6. 6. The method of claim 1, wherein the multi-frame acknowledgement comprises: maintaining a support counter for the candidate track pairs; When the candidate track pair simultaneously satisfies that the square mahalanobis distance is not more than And cosine distance is not greater than When the support counter is incremented; when not satisfied, the support counter decrements; And when the support counter reaches a preset confirmation threshold M, confirming that the candidate track pairs belong to the same physical fruit instance and executing TID merging.
7. 7. The method of claim 1, wherein the confidence weighted sum S is an accumulated sum of confidence scores detected for each TID associated with each frame that is merged into the same fruit instance, and wherein the logical ID associated with the fruit instance is identified as a valid target when the S exceeds a predetermined threshold of confidence.
8. 8. The method according to claim 1, wherein mapping the defect mask bound to the same fruit object to the local coordinate system of the fruit object and rasterizing to a fixed size nxn binary feature grid comprises: And carrying out normalized coordinate transformation on the pixel points of the defect mask by taking the upper left corner of the fruit target bounding box as an origin and the width and the height of the bounding box as dimensions, cutting off the pixel points to a [0,1] interval, and gridding the normalized defect mask into an N multiplied by N binary grid by adopting a nearest neighbor mode.
9. 9. The method of claim 1, wherein the signature cross-over ratio is a cross-over ratio between two of the N x N binary grids, the cross-over ratio between the signatures being not less than a signature threshold When the defect observation under different visual angles is related to the same physical defect, including; When the cross ratio between two N x N binary grids is not less than the signature threshold And when the corresponding defect observation is judged to belong to the same physical defect instance.
10. 10. The method of claim 1, wherein the selection of the representation QTID further comprises a deterministic rule, the method further comprising: When the historic maximum defect areas of a plurality QTID of the same physical defect example are the same, QTID having a longer existing frame number is selected as the representative QTID.
11. 11. The method of claim 1, wherein the cumulative defect area is a sum of historical maximum defect areas each representing QTID under the logical ID, and the control command includes a classification result and spatial location information or time location information corresponding to the logical ID.
12. 12. A hierarchical association tracking and defect quantification system for a rotating spherical fruit target, comprising: the image acquisition unit is used for acquiring a video sequence of the detection area of the roller conveyor belt; The detection and segmentation unit is used for carrying out target detection and instance segmentation on each frame of image of the video sequence to obtain a boundary frame of a parent fruit target and a pixel mask of a child defect target, and obtaining respective confidence scores; The hierarchical binding unit is used for binding the defect mask to the fruit target to which the defect mask belongs according to the spatial inclusion relation between the defect mask and the fruit target boundary frame, and establishing a father-son hierarchical data structure of a fruit track identifier TID and a defect track identifier QTID; The fruit track association unit is used for performing track association on the fruit targets, and establishing a Kalman filtering prediction model for each fruit track, calculating the square Markov distance between the current observation and the prediction, and gating a threshold value with the motion consistency Comparing to screen candidate tracks, extracting feature vectors of fruit appearance ReID, updating appearance prototype vectors by exponential sliding average, calculating cosine distance between appearance prototype vectors, and gating threshold value with appearance consistency The method comprises the steps of comparing to screen candidate tracks, carrying out multi-frame confirmation on candidate track pairs which are simultaneously gated by the motion consistency gate and the appearance consistency gate, weighting the confidence degrees of a plurality of TIDs belonging to the same physical fruit instance to the sum S of the existence degrees after the confirmation, confirming that the fruit instance is an effective target when the S exceeds a preset threshold value, and distributing a unique logic ID; A defect de-duplication unit for performing topological de-duplication on a defect target by mapping a defect mask bound to the same fruit target to a local coordinate system of the fruit target and rasterizing the defect mask into a binary feature grid with a fixed size of N x N, and calculating a signature cross-over ratio for candidate defects, wherein the signature cross-over ratio is not less than a signature threshold value The method comprises the steps of combining defect observations under different view angles into the same physical defect, selecting QTID with the largest historical defect area as a representative QTID of the physical defect for a defect observation set judged to be the same physical defect, establishing alias mapping to enable other QTID in the set to point to the representative QTID, and performing union deduplication on a defect list based on the representative QTID and attributing the representative QTID to the logical ID when TID combination occurs; the grading decision unit is used for counting the accumulated defect areas of all the representatives QTID under the logic ID and comparing the accumulated defect areas with a preset grading threshold value to obtain a grading result; And the control output unit is used for generating a control instruction from the grading result and sending the control instruction to the PLC or the executing mechanism so as to finish sorting execution.
13. 13. The system of claim 12, wherein the fruit track association unit is further configured to time gate screening candidate track pairs prior to track merging and gate a threshold based on the squared mahalanobis distance and motion consistency And the cosine distance and appearance consistency gating threshold value Joint gating is performed.
14. 14. The system of claim 12, wherein the defect deduplication unit is further configured to maintain a historical maximum defect area for each QTID for the same physical defect instance, and to direct other QTID within the same physical defect instance to the representation QTID via alias mapping with QTID having the greatest historical maximum defect area as the representation QTID.

Description

Hierarchical association tracking and defect quantifying method and system for rotating spherical fruit target Technical Field The invention relates to the technical fields of computer vision, pattern recognition, intelligent agricultural equipment and industrial automation control, in particular to a hierarchical association tracking and defect quantifying method and system for a rotary spherical fruit target. Background With the development of intelligent agriculture and intelligent manufacturing technology, the requirements of fruit and vegetable post-processing links on automatic sorting and appearance quality detection are increasingly increased. In the traditional sorting production line, the appearance defects (such as surface scars, decay, stabs and the like) of spherical fruits often depend on manual visual screening, and the efficiency and consistency are limited. Unlike rigid parts with stable postures, spherical fruits generally present a composite motion of translation accompanied by random spin and micro-bouncing on a roller conveyor belt, so that the imaging appearance texture changes nonlinearly with time, defects are positioned on a spherical curved surface, and the projection shape of the defects generates obvious perspective distortion with rotation. The existing DeepSORT method based on Kalman filtering and the association strategy based on the cross-correlation ratio IoU still have the following problems when the scene is applied: (1) The track model is difficult to adapt to compound motion to cause track fragmentation, the prior art adopts a linear constant speed model, the adaptability to appearance mutation caused by spin is insufficient, a target is easy to be lost at a texture mutation or short-time shielding position and is misjudged as a new target, the track ID is frequently switched, and the history detection data of the same fruit is difficult to continuously accumulate. (2) The defect repetition count is caused by the fact that the rigidity IoU is matched with intolerance to spherical perspective distortion, namely, the defect is easy to generate shape compression and stretching when moving from the center to the edge along with the rotation of the fruit, the standard IoU is difficult to identify defect observation with large geometric difference and topological homology, the same physical defect is wrongly recorded into a plurality of independent defect IDs, the defect area statistics is high, and the misclassification is further caused. (3) The simple counting logic has weak anti-interference capability, so that a grading result shakes, namely, factors such as motion blur, uneven illumination, dust interference and the like exist in an industrial field, false detection with low confidence coefficient is easy to generate, and if the noise and the effective detection are given the same weight, judgment shake is easy to occur near a grading threshold value, so that the equipment repetition precision is reduced. Therefore, a technical scheme capable of adapting to the spin movement of the spherical fruit, realizing the stable association of the track and performing the de-weighting on the spherical defect is needed, so as to improve the consistency and the robustness of defect statistics and graded output. Disclosure of Invention The invention aims to solve the problems of track fragmentation, defect repetition count, classification result jitter and the like in a self-rotating spherical fruit scene of a roller conveyor belt in the prior art, and provides a classification association tracking and defect quantifying method and system for a rotating spherical fruit target, so as to realize stable tracking of the same fruit instance, de-duplication metering of the same physical defect and stable output of classification results. In order to achieve the above purpose, the present invention provides the following technical solutions. 1. Method of The invention provides a hierarchical association tracking and defect quantifying method for a rotary spherical fruit target, which comprises the following steps: Step S1, multitasking sensing and hierarchical binding. The method comprises the steps of obtaining a video sequence of a detection area of a roller conveyor belt, carrying out target detection and instance segmentation on each frame image of the video sequence to obtain a boundary frame of a parent fruit target and pixel masks of child defect targets, obtaining respective confidence scores, binding the defect masks to the fruit targets to which the defect masks belong according to the spatial inclusion relation between the defect masks and the fruit target boundary frame, and establishing a parent-child hierarchical data structure of a fruit track identifier TID and a defect track identifier QTID. And S2, associating the fruit track with the validity confirmation VisionLock. Establishing a Kalman filtering prediction model for each fruit track, calculating the square Markov dista