CN-121982280-A - Mobile robot pallet identification method based on computer vision

CN121982280ACN 121982280 ACN121982280 ACN 121982280ACN-121982280-A

Abstract

The invention discloses a mobile robot pallet identification method based on computer vision, which relates to the technical field of robot vision perception and comprises the steps of acquiring camera internal parameters and camera external parameters from a robot base, collecting color images and depth images, performing de-distortion and alignment treatment, generating a near-earth mask based on fitting a ground plane by the depth images, constructing a structure dimension candidate region by combining depth gradients and brightness gradients, extracting an effective structure, weighting and solving pallet center pose and pose covariance, further calculating docking deviation and constructing an error uncertainty domain to complete docking judgment, improving structural identification accuracy by means of spatial constraint and multi-source information fusion, enhancing pose estimation stability by means of observation quality weighting and uncertainty quantification, realizing docking risk conservation assessment, and improving docking safety and reliability.

Inventors

CHEN JIAN
XU WEITING
PANG WENYAO

Assignees

浙江科聪控制技术有限公司

Dates

Publication Date: 20260505
Application Date: 20260408

Claims (10)

1. A mobile robot pallet identification method based on computer vision, comprising: acquiring an internal parameter of a camera and an external parameter from the camera to a robot base; collecting color images and depth images, and performing de-distortion and alignment treatment; Fitting a ground plane based on the depth image, and generating a ground-near mask; based on the near-earth mask, combining the depth gradient and the brightness gradient, constructing a structure dimension candidate region of a left fork hole, a right fork hole and a front edge guide edge of the pallet; Generating visibility probability and boundary segment confidence in each structure dimension candidate area, extracting observable boundary segments, calculating structure visibility probability and boundary coverage rate, and determining an effective structure set; Calling a pallet structure template, carrying out normalized soft corresponding matching on observation boundary points of an effective structure, and constructing a geometric residual observation set; According to the structure visibility probability and the boundary coverage rate, weighting and solving the central pose from the pallet to the camera, and calculating the pose covariance of the pallet relative to the camera; Transforming the central pose to a robot base coordinate system, and calculating the transverse deviation, the height deviation and the yaw angle deviation between the center line of the fork hole and the center line of the fork; Based on the lateral deviation, the height deviation and the yaw angle deviation, an error uncertainty domain is constructed by combining the pose covariance, the most unfavorable docking margin is calculated, and the docking judgment of the robot on the pallet is output.
2. The computer vision based mobile robot pallet identification method of claim 1, wherein the obtaining camera intrinsic and camera-to-robot base extrinsic comprises: setting a plane calibration plate in a robot observability area, and collecting multi-frame calibration images of the plane calibration plate at different distances, different visual angles and different postures; extracting the pixel coordinates of the characteristic points in each frame of calibration image, and establishing the three-dimensional coordinates of the characteristic points under the coordinate system of the calibration plate; constructing a re-projection error between the three-dimensional coordinates and the pixel coordinates, and solving a camera focal length parameter, a principal point parameter and a distortion parameter by minimizing the re-projection error to obtain a camera internal reference matrix; Under the condition that the plane calibration plate is opposite to the base pose of the robot, solving the pose of the calibration plate opposite to the camera by adopting a PnP algorithm based on the three-dimensional coordinates of the characteristic points and the corresponding pixel coordinates; And according to the coordinate transformation link relation, performing matrix operation on the pose from the calibration plate to the base and the solving pose from the calibration plate to the camera, and obtaining an external parameter matrix from the camera to the robot base.
3. The computer vision based mobile robotic pallet identification method of claim 1, wherein the performing de-distortion and alignment process comprises: establishing a de-distortion lookup table based on internal parameters and distortion parameters of the color channel and the depth channel; Resampling the color image and the depth image by using a de-distortion lookup table to obtain a de-distortion image; And back-projecting the effective depth pixels in the depth image into three-dimensional points, and projecting the three-dimensional points to a color image coordinate system according to the external reference relation from the depth camera to the color camera to perform space alignment of the depth image and the color image.
4. The computer vision based mobile robotic pallet identification method of claim 1, wherein the generating a near-earth mask comprises: Selecting candidate pixels in the lower half area of the depth image; Back-projecting the candidate pixels into a three-dimensional point set; performing plane fitting by adopting a random sampling consistency method, and selecting a plane with the largest number of inner points as an initial ground plane; Carrying out least square finish on an initial ground plane, and carrying out orientation constraint verification based on an included angle between a plane normal direction and a vertical direction; And generating a near-earth mask according to the distance threshold value from the depth point to the ground.
5. The computer vision based mobile robotic pallet identification method of claim 1, wherein constructing the structural dimension candidate regions of the left fork aperture, the right fork aperture, and the leading edge guide edge of the pallet comprises: Respectively calculating depth gradient amplitude and brightness gradient amplitude in the near-earth effective pixel set; Determining pixels with depth gradient amplitude values larger than or equal to a depth threshold value as a depth boundary zone; Determining pixels with brightness gradient amplitude values larger than or equal to a brightness threshold value as texture boundary bands; taking the intersection of the depth boundary band and the texture boundary band as a gradient consistency candidate band; and constructing geometric windows of the left fork hole, the right fork hole and the leading edge guide edge based on the gradient consistency candidate belt, and taking intersection sets to form candidate areas of each structural dimension.
6. The computer vision based mobile robotic pallet identification method of claim 5, wherein the generating a visibility probability and a boundary segment confidence comprises: clipping the structure dimension candidate region into a fixed resolution image block; Inputting the resolution image block and the corresponding mask block into a joint characteristic network sharing a backbone network and a double-output-head structure; generating a pixel level visibility probability field through a visibility output header; and generating a boundary segment confidence field in multiple discrete directions through the boundary output head.
7. The computer vision based mobile robotic pallet identification method of claim 6, wherein constructing the set of geometric residual observations comprises: uniformly sampling an observable boundary segment of an effective structure to obtain an observed boundary point; Mapping the observation boundary points to normalized candidate region coordinates, calculating Euclidean distance between the observation boundary points and the template boundary points, and converting the Euclidean distance into similarity weights; Normalizing the similarity weight to form a soft corresponding weight matrix; and constructing a geometric residual error set based on the projection positions of the template three-dimensional structure points and the distances between the soft corresponding target points.
8. The computer vision based mobile robot pallet identification method of claim 7, wherein the weighting the central pose of the pallet to camera comprises: Carrying out the parametric expression of the Liu group by adopting six-dimensional minimum parameters to the pose, and constructing a weighted objective function comprising structural weights and a robust cost function; carrying out optimization solution on the weighted objective function by adopting an iterative re-weighted least square algorithm; After iteration convergence, constructing an information matrix based on the jacobian matrix and the weight matrix; and inverting the information matrix to obtain the pose covariance matrix.
9. The computer vision based mobile robotic pallet identification method of claim 8, wherein calculating the lateral deviation, the height deviation, and the yaw angle deviation between the fork aperture centerline and the fork centerline comprises: Converting the central pose of the pallet to a base coordinate system through external parameters from the camera to the robot base; respectively constructing a fork hole center line and a fork center line under a base coordinate system, calculating offset of the fork hole center line relative to the fork center line in the horizontal lateral direction as transverse deviation, calculating a difference value between the height of the fork hole center and a fork height reference as height deviation, and calculating an included angle of the two center lines in the horizontal plane as yaw angle deviation.
10. The computer vision based mobile robot pallet identification method of claim 9, wherein the output robot-to-pallet docking determination comprises: Mapping the pose covariance into a docking error covariance matrix based on first-order linearization of the central pose parameter; Constructing a three-dimensional ellipsoid error uncertainty domain under a confidence level, and respectively calculating the least favorable absolute value upper bound of the transverse deviation, the height deviation and the yaw angle deviation in the three-dimensional ellipsoid error uncertainty domain; comparing the most unfavorable absolute value upper bound with a corresponding allowable tolerance, and taking the minimum value of the residual margin proportion as the most unfavorable butting margin; And outputting the butt joint judgment according to the most unfavorable butt joint margin.

Description

Mobile robot pallet identification method based on computer vision Technical Field The invention relates to the technical field of robot vision perception, in particular to a mobile robot pallet identification method based on computer vision. Background With the development of intelligent logistics and warehouse automation, mobile robots are increasingly widely applied to cargo handling, pallet butt joint and autonomous operation. In the prior art, a method based on two-dimensional image feature extraction is generally adopted for pallet identification, target positioning is realized by combining edge detection, shape matching or feature point matching, a part of schemes are further introduced into a depth sensor, and three-dimensional pose information of the pallet is obtained through RGB-D data fusion. In engineering practice, the estimation of pallet fork hole position and posture parameters is often realized by preprocessing images, screening candidate areas, fitting geometric models and the like, so that a posture basis is provided for the robot to execute docking control. However, the conventional method still has certain limitations under the complex working environment, namely firstly, under the conditions of strong shielding, illumination change or ground interference, the conventional method is only dependent on a detection mode of local edges or characteristic points, so that the visibility of the structure is easy to fluctuate, and the pose estimation stability is influenced, secondly, in the pose solving process, most methods do not carry out explicit modeling and quantification on the uncertainty of observation, and error propagation and safety margin are difficult to comprehensively consider in the phase of butt joint judgment, so that the adaptability to a high-precision butt joint scene is limited to a certain extent. Disclosure of Invention The present invention has been made in view of the above-described problems occurring in the prior art. Therefore, the invention provides a mobile robot pallet recognition method based on computer vision, which solves the problems of great fluctuation of structural visibility due to environmental influence and lack of quantitative analysis of pose uncertainty in the prior art. In order to solve the technical problems, the invention provides the following technical scheme: The invention provides a mobile robot pallet identification method based on computer vision, which comprises the steps of obtaining camera internal parameters and camera-to-robot base external parameters, collecting color images and depth images, carrying out de-distortion and alignment processing, generating a near ground mask based on the depth images, constructing structure dimension candidate areas of left fork holes, right fork holes and front edge guide edges of a pallet based on the near ground mask and combining depth gradients and brightness gradients, generating visibility probability and boundary segment confidence in each structure dimension candidate area, extracting observable boundary segments, calculating structure visibility probability and boundary coverage rate, determining an effective structure set, calling a pallet structure template, carrying out normalized soft corresponding matching on observation boundary points of the effective structure, constructing a geometric residual error observation set, weighting and solving a central pose of the pallet to the camera according to the structure visibility probability and the boundary coverage rate, calculating the pose covariance of the pallet relative to the camera, converting the central pose to a robot base coordinate system, calculating the transverse deviation and the height deviation between a fork hole center line and a pallet center line, calculating the transverse deviation and a height deviation and a pallet deviation, and determining the transverse deviation and the maximum deviation and not combining the transverse deviation and the maximum deviation. The method comprises the steps of setting a plane calibration plate in a robot observability area, collecting multi-frame calibration images of the plane calibration plate at different distances, different visual angles and different postures, extracting characteristic point pixel coordinates in each frame of calibration image, establishing three-dimensional coordinates of characteristic points in a calibration plate coordinate system, constructing a re-projection error between the three-dimensional coordinates and the pixel coordinates, solving a camera focal length parameter, a main point parameter and a distortion parameter through minimizing the re-projection error, obtaining a camera internal parameter matrix, solving the pose of the calibration plate relative to a camera based on the characteristic point three-dimensional coordinates and the corresponding pixel coordinates under the condition that the plane calibration plate is relative to the robot base