US-12622357-B2 - Method for controlling motion parameters of pepper harvester based on combination of point clouds and images

US12622357B2US 12622357 B2US12622357 B2US 12622357B2US-12622357-B2

Abstract

A method for controlling motion parameters of a pepper harvester based on combination of point clouds and images, including: (1) performing space-time calibration on a radar and a camera, constructing a data collection platform, and pre-processing image and point cloud data; (2) acquiring a real-time number of pepper in front of a harvester and an average height from canopies of pepper plants to soil through a point cloud, acquiring a cutting width and a pepper fruit proportion in front of the harvester through an image, and acquiring a comprehensive lowermost location of pepper fruits by combination of the point cloud and the image; (3) acquiring real-time motion parameters of the pepper harvester; and (4) controlling a real-time rotation speed, operation speed, and height of a drum using a fuzzy PID control method based on the comprehensive lowermost location of the pepper fruits and a predicted value of a feeding quantity.

Inventors

Xinyan QIN
Jin LEI
Chenming CHENG
Xinyu Zhang
Bingpeng WANG
Wenxing JIA
Zhiyuan ZHAI

Assignees

SHIHEZI UNIVERSITY

Dates

Publication Date: 20260512
Application Date: 20231130
Priority Date: 20230616

Claims (10)

1 . A method for controlling motion parameters of a pepper harvester based on combination of point clouds and images, comprising: acquiring laser point cloud data and image data of pepper plants in a to-be-harvested region from a lidar and a camera fixedly mounted on a cab top of a harvester; implementing time synchronization between the camera and the lidar using a multi-source sensor time synchronization method based on frequency self-matching and then implementing space synchronization using a combined calibration method on a basis of the time synchronization so as to synchronize the laser point cloud data and the image data in both time and space thereafter, performing pre-processing operations comprising denoising and enhancing on the laser point cloud data and the image data collected by the camera, reducing an area of a non-operation region, reducing noise, increasing a running speed of a system, performing coordinate system transformation on the laser point cloud data collected by the lidar so as to obtain same region of interest (ROI) of processed laser point cloud data and processed image data, and finally obtaining a conversion relationship between an image pixel coordinate system and a lidar coordinate system, a conversion relationship between the lidar coordinate system and a vehicle coordinate system, and a conversion relationship between the vehicle coordinate system and a geodetic coordinate system; acquiring a real-time number n of the pepper plants in the to-be-harvested region in front of the pepper harvester and an average height L from canopies of the pepper plants to soil through the processed laser point cloud data; acquiring a real-time cutting width w and a pepper fruit proportion p of the to-be-harvested region in front of the pepper harvester through the processed image data; acquiring a comprehensive lowermost location of pepper fruits in the to-be-harvested region by combination of the point clouds and images; measuring a real-time rotation speed N of a picking drum using a Hall sensor, symmetrically and fixedly mounting sensor probes on left and right ends of a drum shaft, fixing steel magnets on spring-tooth mounting plates on left and right sides, measuring a real-time operation speed V 0 of the pepper harvester by a navigation speed measurement system, mounting the navigation speed measurement system on the cab top of the pepper harvester, and mounting drum height sensors on both sides of a drum supporting shaft for acquiring a real-time height of the drum, the drum height sensors being ultrasonic sensors; predicting a pepper plant density q through an improved back propagation (BP) neural network model, taking the real-time number n of the pepper plants, a height H of a cutting table, the average height L from the canopies of the pepper plants to the soil, and the pepper fruit proportion p as inputs of an improved BP neural network and a total mass m under a fixed area as an output of the improved BP neural network, calculating the pepper plant density q through a formula according to the total mass m, and further calculating a predicted value of a feeding quantity at a next time, a formula for calculating the pepper plant density being: q = m 1000 · s ′ . wherein m is the total mass under the fixed area predicted by the improved BP neural network, and s t is an actual area processed; a formula for calculating the feeding quantity being: Q = wvq wherein Q is the predicted value of the feeding quantity, w is the real-time cutting width, v is an operation speed of the pepper harvester, and q is the pepper plant density; and controlling a real-time rotation speed of the drum and the operation speed of the pepper harvester according to the predicted value of the feeding quantity and controlling the height of the cutting table according to the comprehensive lowermost location of the pepper fruits, based on fuzzy proportional-integral-derivative (PID) control.
2 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the lidar, the camera, and the navigation speed measurement system are a solid-state lidar, a drive-free universal serial bus (USB) high-definition camera, and a Beidou Navigation Satellite System/Global Position System dual-system navigation speed measurement system, the navigation speed measurement system being a real-time speed acquisition method based on extended Kalman filtering so as to improve a measurement accuracy and noise suppression capability of speed measurement.
3 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the implementing the time synchronization and the space synchronization between the lidar and the camera further comprises: implementing the time synchronization using the multi-source sensor time synchronization method based on the frequency self-matching and then implementing the space synchronization using the combined calibration method on the basis of the time synchronization, wherein the vehicle coordinate system is to be defined during calibration, and laser point cloud coordinates are obtained through image pixel coordinates and corresponding geodetic coordinates are obtained through the laser point cloud coordinates finally.
4 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the acquiring of the real-time number n of the pepper plants in the to-be-harvested region in front of the pepper harvester and the average height L from the canopies of the pepper plants in the to-be-harvested region to the soil further comprises: acquiring the real-time number n of the pepper plants in the to-be-harvested region and the average height L from the canopies of the pepper plants in the to-be-harvested region to the soil by performing a Euclidean distance clustering segmentation algorithm and a Density-Based Spatial Clustering of Applications with Noise (DBSCAN)-based adaptive point cloud clustering on the processed laser point cloud data, respectively, for predicting the pepper plant density q by the improved BP neural network model subsequently.
5 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the acquiring of the real-time cutting width w of the pepper harvester further comprises: collecting and processing pictures of the pepper plants in front of the cutting table, marking the cutting table of the pepper harvester in the pictures thereafter, performing spacing template matching once to mark a row of plants nearest to the cutting table, taking a length of the cutting table as a total cutting width to further determine a number of plants within the cutting table in front of the cutting table, obtaining a distance between two farthest plants at this moment, and then obtaining the real-time cutting width w.
6 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the acquiring of the pepper fruit proportion p of the pepper plants in the to-be-harvested region further comprises: processing the images collected by the camera to obtain a binarized image of the pepper fruits and a binarized image of a whole pepper plant, and then acquiring a number of pixel points of the binarized image of the pepper fruits and a number of pixel points of the binarized image of the whole pepper plant by an iterator access, a ratio of the number of pixel points of the binarized image of the pepper fruits to the number of pixel points of the binarized image of the whole pepper plant being the pepper fruit proportion p, the image processing comprising the following steps: performing histogram equalization processing again on a pre-processed picture to further increase a contrast of the picture and a clarity of the image; filtering processing: removing image noise and better preserving image edges using bilateral filtering based on comprehensive consideration; image segmentation: performing image segmentation using an Otsu's Method; morphological processing: eliminating small particle noise in the image using erosion and dilation; and obtaining the binarized image of the pepper fruits and the binarized image of the whole pepper plant, respectively.
7 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the acquiring of the comprehensive lowermost location of the pepper fruits in the to-be-harvested region further comprises: after the pixel points of the binarized image of the whole pepper plant are scanned and detected to obtain a pepper pixel coordinate set of the pepper fruits located at a lowermost layer in a pepper plant image, obtaining a corresponding lidar coordinate point set through a conversion relationship between image pixel coordinates and lidar coordinates, obtaining a corresponding geodetic coordinate point set through a conversion relationship between the lidar coordinates and geodetic coordinates, and finally calculating an average value of Z coordinate values of points in a geodetic coordinate set as the comprehensive lowermost location of the pepper fruits.
8 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the controlling of the operation speed further comprises: setting a reasonable feeding quantity range λ 1 <Q<λ 2 in advance according to a magnitude of a torque to be applied to the drum shaft, appropriately reducing the operation speed when the predicted value of the feeding quantity exceeds the reasonable feeding quantity of the pepper harvester, appropriately increasing the operation speed when the predicted value of the feeding quantity is lower than the reasonable feeding quantity of the pepper harvester, and reducing a failure rate of the pepper harvester while ensuring an operation efficiency of the pepper harvester, a formula for calculating a target operation speed being: V = V 0 + Q rated - Q 2.5 · q wherein V is the target operation speed, V 0 is the real-time operation speed of the pepper harvester, Q rated is a rated feeding quantity of the pepper harvester, Q is the predicted feeding quantity, and q is the pepper plant density.
9 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the controlling the real-time rotation speed of the drum comprises: setting a reasonable feeding quantity range λ 1 <Q <λ 2 in advance according to a magnitude of a torque to be applied to the drum shaft, not changing the real-time rotation speed of the drum when the predicted value is within the reasonable feeding quantity range, appropriately increasing the real-time rotation speed of the drum when the predicted feeding quantity is lower than a threshold, and reducing the real-time rotation speed of the drum conversely, an adjustment range of the real-time rotation speed of the drum being consistent with a change range of the predicted feeding quantity.
10 . The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images according to claim 1 , wherein the controlling the height of the cutting table comprises: presetting a reasonable adjustment range of the height of the cutting table according to a measured average height from the canopies of the pepper plants to the soil, and resizing the cutting table according to the comprehensive lowermost location of the pepper fruits.

Description

CROSS REFERENCE TO RELATED APPLICATION This application is based upon and claims foreign priority to Chinese Patent Application No. 202310714807.0, filed on Jun. 16, 2023, the entire content of which is incorporated herein by reference. TECHNICAL FIELD The present disclosure belongs to the technical field of pepper harvesters, and particularly relates to a method for controlling motion parameters of a pepper harvester based on combination of point clouds and images. BACKGROUND China is the country with the largest planting area of pepper in the world, and pepper is also one of the favorite vegetables for Chinese. The output value and benefit of pepper have been far higher than other vegetables, and pepper ranks first among vegetables. However, due to the particularity of the growth and mechanized harvesting of pepper, it is a great challenge to match motion parameters of a pepper harvester. At present, the intelligence maturity of the pepper harvester is not high, and the running state of the machine is determined only by manual experience. Therefore, the operation efficiency is low, and it is difficult to realize the reasonable matching of external load and power, which is not conducive to the intelligent development of the pepper harvester. The motion parameters of the pepper harvester are closely related to the density of pepper plants, an operation width, the height of a drum, a pepper fruit proportion, and a comprehensive lowermost location of pepper fruits. Excessive density and large operation width will overload key parts of the pepper harvester, and cause a series of problems such as drum blockage, increase of pepper breakage rate, reduction of net harvest rate, and difficulty of harvester driving. In view of the above problems, the existing technology mainly analyzes the magnitude of a force applied to a key part of the pepper harvester so as to adjust the motion parameters of the harvester, for example, analyzes the magnitude of a torque applied to a drum shaft and the magnitude of an impact force applied to spring teeth. However, this adjustment is carried out under the overload situation, which has serious lag, single control, and low intelligence, thus causing insufficient research on intelligent identification and intelligent prediction of the motion parameters of the pepper harvester. The present disclosure provides a method for controlling motion parameters of a pepper harvester based on combination of point clouds and images. The method can obtain the number of pepper plants in a to-be-harvested region in front of the pepper harvester, an average distance from canopies of the pepper plants to soil, a pepper fruit proportion, an operation width, and a comprehensive lowermost location of pepper fruits through images and point clouds. A real-time feeding quantity of a drum is predicted according to the pepper plant parameters, and real-time motion parameters of the pepper harvester are obtained in combination with a Hall sensor, a speed measurement sensor, and a drum height sensor. This method can control the operation speed of the harvester, the rotation speed of the drum, and the height of the drum in real time, thus avoiding the problem of serious lag, significantly improving the net rate of harvesting pepper and reducing the breakage rate of the pepper harvester, reducing the operation strength of an operator of the pepper harvester, and conforming to the development trend of intelligent agricultural machinery. SUMMARY In order to solve the above-mentioned technical problems, the present disclosure provides a method for controlling motion parameters of a pepper harvester based on combination of point clouds and images, which may predict a feeding quantity of a harvester at a next time by combination of images and point clouds, acquire the growth of pepper, and the distribution of pepper, and then control a rotation speed of a drum, an operation speed of the pepper harvester, and the height of a cutting table in real time. The method reasonably matches a power output of a hydraulic system of the harvester and an external load, thus improving the real-time performance, increasing the net harvest rate of pepper, reducing the damage rate of pepper and the failure rate, realizing automatic control, and greatly improving the intelligence level. The present disclosure is implemented by the following technical solutions. The present disclosure provides a method for controlling motion parameters of a pepper harvester based on combination of point clouds and images. The method for controlling motion parameters of a pepper harvester based on combination of point clouds and images includes the following steps: S1: acquiring laser point cloud data and image data of pepper plants in a to-be-harvested region from a lidar and a camera fixedly mounted on a cab top of a harvester;S2: implementing time synchronization between the camera and the lidar using a multi-source sensor time synchronization method b