CN-122023514-A - Portal crane locking cylinder locking hole accurate alignment method and system based on point cloud and visual depth fusion

CN122023514ACN 122023514 ACN122023514 ACN 122023514ACN-122023514-A

Abstract

The invention provides a method and a system for accurately aligning a lock hole of a gantry crane lock column based on point cloud and visual depth fusion, and belongs to the technical field of intelligent port automation. The method comprises the steps of performing off-line calibration on an RGB camera, performing off-line joint calibration on a laser radar-RGB camera, fusing two-dimensional images and three-dimensional point cloud data, performing lock column and lock hole visual segmentation identification based on YOLOv models, projecting a two-dimensional detection result to a three-dimensional point cloud space, performing lock column and lock hole positioning based on clustering and geometric fitting, and performing lock column and lock hole deviation calculation and self-adaptive alignment control. The system comprises an RGB camera, a laser radar, a GPU computing platform and a PLC control platform. The invention realizes high-precision, high-efficiency and high-reliability automatic alignment through the full-flow closed-loop design of point cloud and vision fusion.

Inventors

LI HUACHAO
LIU HUATING
GAN ZHIJIE
QI LIANGJIAN
WU YONG
ZHOU KE

Assignees

苏州物量智能科技有限公司

Dates

Publication Date: 20260512
Application Date: 20251230

Claims (10)

1. The accurate alignment method for the locking hole of the gantry crane locking column based on the fusion of the point cloud and the visual depth is characterized by comprising the following steps of: step S1, performing off-line calibration of an RGB camera, namely performing internal reference calibration and distortion correction on the RGB camera based on a standard checkerboard calibration plate to obtain a camera internal reference matrix and a distortion coefficient vector; s2, performing off-line joint calibration on the laser radar and the RGB camera based on a standard checkerboard calibration plate to obtain an external parameter transformation matrix; step S3, the two-dimensional image and the three-dimensional point cloud data are fused, wherein in the operation process of the gantry crane, an RGB camera and a laser radar synchronously acquire the two-dimensional image and the three-dimensional point cloud data of a container operation area, distortion correction is carried out on the two-dimensional image based on a camera internal reference matrix and an external reference transformation matrix, and space-time alignment and fusion are carried out on the two-dimensional image and the three-dimensional point cloud data, so that a distortion correction image and fused three-dimensional point cloud data are obtained; S4, carrying out lock column and lock hole visual segmentation recognition based on YOLOv model, namely adopting a trained YOLOv model to recognize and position lock column and lock hole positions of the container from the distortion correction image in real time; Step S5, projecting a two-dimensional detection result to a three-dimensional point cloud space, namely, based on the detected positions of the lock posts and the lock holes, combining three-dimensional cloud data of fusion points, projecting the two-dimensional detection result to the three-dimensional space to obtain a preliminary projection point cloud, wherein the preliminary projection point cloud comprises a lock post point cloud set and a lock hole point cloud set; s6, carrying out lock cylinder and lock hole positioning based on clustering and geometric fitting, namely extracting three-dimensional positions and postures of lock cylinders and lock holes from the primary projection point cloud by adopting a point cloud clustering and geometric fitting algorithm; And S7, calculating the deviation between the lock column and the lock hole and performing self-adaptive alignment control, namely calculating three-dimensional space deviation based on the three-dimensional positions and the postures of the lock column and the lock hole, including the position deviation and the posture deviation, generating an alignment control instruction, and executing the alignment control instruction.
2. The accurate alignment method for the locking hole of the gantry crane locking column based on the fusion of the point cloud and the visual depth of the invention according to claim 1, wherein the specific process of the step S1 is as follows: Step S11, aiming at a standard checkerboard calibration plate, collecting RGB camera images of different angles and different space placement positions in a calibration scene; Step S12, automatically detecting angular points in the checkerboard by using a cv2.findCHessboard Corders () function of the OpenCV library, and optimizing sub-pixel precision by using the cv2.corerSubPix () function, wherein the angular point detection precision reaches 0.1 pixel; Step S13, solving a camera internal reference matrix and a distortion coefficient vector through a cv2.calibrecat eParamera () function; step S14, performing distortion correction on a plurality of original images during gantry crane operation shot by a wide-angle lens of an RGB camera by using a cv2.undistitor () function; Step S15, calculating the re-projection error of each calibrated image after calibrating a plurality of original images, carrying out statistical analysis and verification, and outputting a final camera internal reference matrix and a distortion coefficient vector, wherein the final camera internal reference matrix and the distortion coefficient vector are expressed as: , distortion coefficient vector dist= [ k1, k2, p1, p2, k3], Wherein fx and fy are focal lengths of the RGB camera in x and y directions, and the unit is a pixel; cx and cy are the main point coordinates of the image; k1, k2 and k3 are radial distortion coefficients; p1 and p2 are tangential distortion coefficients.
3. The accurate alignment method for the locking hole of the gantry crane locking column based on the fusion of the point cloud and the visual depth of the invention according to claim 1, wherein the specific process of the step S2 comprises the following steps: S21, multi-position multi-pose data acquisition, namely placing standard checkerboard calibration plates with known sizes at least at 2 different calibration space positions in a calibration scene, wherein the standard checkerboard calibration plates under each calibration space position show different translation poses and rotation poses; Synchronously acquiring RGB camera images and laser radar 3D point cloud data for each calibration space position; S22, collecting image feature points, namely detecting internal corner points of a checkerboard by using a cv2.findCHessboard Corders () function on an collected RGB camera image to obtain a 2D image corner pixel coordinate set; sub-pixel precision optimization is performed on the detected 2D image corner points by using a cv2.corersubPix () function; Repeating the above process for the calibration images of the plurality of calibration space positions, and accumulating the pixel coordinate data of the 2D image corner points of all the calibration space positions; S23, collecting point cloud characteristic points, namely denoising collected laser radar 3D point cloud data and outliers; extracting the point cloud of the plane of the calibration plate by a point cloud segmentation algorithm, and fitting the plane of the calibration plate by using a RANSAC plane fitting algorithm; extracting 3D coordinates of the 3D point cloud corner points corresponding to the 2D image corner points in the calibration plate plane point cloud according to the geometric structure of the checkerboard and the point cloud density distribution; Scaling correction is carried out on 3D coordinates of the 3D point cloud corner points according to the actual physical size of the checkerboard; repeating the process for the laser radar 3D point cloud data of a plurality of calibration space positions, and accumulating the 3D coordinate data of the 3D point cloud points of all the calibration space positions; Step S24, point pair matching and verification, namely verifying whether the number of the extracted 2D image corner points is consistent with the number of the 3D point cloud corner points or not for each calibration space position, and prompting errors and re-acquiring data if the number of the extracted 2D image corner points is inconsistent with the number of the 3D point cloud corner points; Establishing a one-to-one correspondence between 2D image corner points and 3D point cloud corner points to form a point-to-data set; Step S25, constructing a nonlinear optimization objective function, wherein the optimization objective function is defined as: minΣ||pi-K·(R·Pi+t)||2, where pi is the 2D image corner pixel coordinates, Pi is the corresponding 3D point cloud corner 3D coordinates, K is a calibrated camera internal reference matrix, R and t are the extrinsic transformation matrix to be solved; carrying out nonlinear least square optimization by using a Powell optimization algorithm or a Levenberg-Marquardt algorithm, and iteratively solving optimal rotation and translation parameters; Setting the initial guess value as zero rotation and zero translation or obtaining a rough initial value through a PnP algorithm, automatically searching a global optimal solution by an optimization algorithm, outputting an optimal external parameter transformation matrix after optimization convergence, and representing as follows: , Wherein the upper left 3 x 3 sub-matrix is a rotation matrix R; The upper right 3 x 1 column vector is the translation vector t= [ tx, ty, tz ] T .
4. The accurate alignment method of a gantry crane locking hole based on point cloud and visual depth fusion according to claim 1, wherein in the step S3, distortion correction is performed on a two-dimensional image, space-time alignment and fusion are performed on the two-dimensional image and three-dimensional point cloud data, and the specific processes of obtaining a distortion correction image and fusing the three-dimensional point cloud data are as follows: Step S31, time synchronization and distortion correction, namely aligning two-dimensional images and three-dimensional point cloud data through a hardware trigger signal or a software time stamp; Carrying out distortion correction on the acquired two-dimensional image by using a camera internal reference matrix and a distortion coefficient vector to obtain a distortion correction image; step S32, mapping point cloud to image projection, namely carrying out coordinate system transformation on the acquired laser radar 3D point cloud data through an external parameter transformation matrix, and converting the 3D point cloud points from a laser radar coordinate system to a camera coordinate system; Using a camera internal reference matrix, projecting 3D point cloud points under a camera coordinate system to a 2D image plane through a cv2.projectpoints () function, and processing shielding conditions when a plurality of 3D point cloud points are projected to the same image point through a depth buffer algorithm; And (3) generating enhanced point clouds with color attributes for RGB color information of corresponding image points on the two-dimensional image in a manner of associating each 3D point cloud point, and simultaneously associating depth information of corresponding 3D point cloud points for each image point.
5. The accurate alignment method of the gantry crane locking hole based on the point cloud and visual depth fusion of claim 1, wherein the model YOLOv training process in the step S4 is specifically as follows: s41, YOLOv, training and deploying a model, namely collecting lock column and lock hole images of containers with different illumination conditions, different angles and different types, establishing a training dataset, and carrying out pixel-level labeling; Using YOLOv-seg architecture, training model based on COCO pre-training weight transfer learning, data enhancement is adopted in the training process, and optimization is performed through TensorRT or ONNX Runtime after training.
6. The accurate alignment method of locking holes of locking columns of a gantry crane based on point cloud and visual depth fusion according to claim 1 or 5, wherein the process of identifying and positioning the positions of locking columns and locking holes of a container from the distortion correction image in real time in step S4 comprises: Step S42, two-dimensional positioning of a lock column and a lock hole, namely inputting a distortion correction image into YOLOv models, and outputting a detection frame containing the lock column and the lock hole, a confidence score and a segmentation mask; And extracting the barycenter coordinates and the characteristic key points of the lock posts and the lock holes through the segmentation mask for each detected lock post and each detected lock hole, and filtering through a confidence threshold value to obtain a detection frame or segmentation mask for finally positioning the lock posts and the lock holes.
7. The accurate alignment method for the locking hole of the gantry crane locking column based on the fusion of the point cloud and the visual depth according to claim 1, wherein the specific process of the step S5 comprises the following steps: step S51, extracting point clouds of the region of interest, namely extracting all point clouds of the corresponding region in the three-dimensional cloud data of the fusion point according to a detection frame or a segmentation mask of a lock column and a lock hole output by a YOLOv model to obtain a preliminary projection point cloud comprising a lock column point cloud set and a lock hole point cloud set; Step S52, preprocessing and filtering the point cloud, namely removing noise points and outliers in the preliminary projection point cloud by using a statistical filter or a radius filter; Removing points exceeding the effective distance range of lock cylinder lock hole alignment operation in the preliminary projection point cloud through depth threshold filtering, and reserving effective point cloud data of the lock cylinder and the lock hole; And reducing the number of the point clouds in the area where the density of the primary projection point clouds exceeds a threshold value by adopting a voxel downsampling method.
8. The accurate alignment method for the locking hole of the gantry crane locking column based on the fusion of the point cloud and the visual depth according to claim 1, wherein the specific process of the step S6 comprises the following steps: Step S61, point cloud clustering segmentation, namely setting a clustering distance threshold, and respectively carrying out Euclidean clustering on a lock column point cloud set and a lock hole point cloud set to separate out point clouds of single lock columns or lock holes; step S62, lock cylinder geometric fitting, namely adopting a RANSAC algorithm to fit a cylinder model, and solving cylinder size parameters, wherein the cylinder size parameters comprise a central axis direction vector, a radius and a height; calculating the three-dimensional coordinates of the center point of the top end or the bottom end of the lock cylinder based on the cylinder size parameters; Calculating the attitude angle of the lock cylinder based on the direction vector of the central axis of the cylinder, wherein the three-dimensional coordinates of the lock cylinder and the central point of the top end or the bottom end of the lock cylinder are used for representing the three-dimensional position and the attitude of the lock cylinder; step S63, performing geometric fitting of the lock hole, namely performing plane fitting on the lock hole point cloud, and extracting a normal vector and a plane equation of a plane where the lock hole is located; identifying a hole area on a fitting plane through point cloud density analysis or an edge detection algorithm, and determining a boundary of a lock hole; calculating the geometric center or the mass center of the lock hole boundary point cloud to obtain the three-dimensional coordinate of the center point of the lock hole; Fitting is carried out by adopting a corresponding geometric model according to the actual shape of the lock hole, and the size parameter and the direction of the lock hole are extracted and used for representing the three-dimensional position and the gesture of the lock hole.
9. The accurate alignment method for the locking hole of the gantry crane locking column based on the fusion of the point cloud and the visual depth according to claim 1, wherein the specific process of the step S7 comprises the following steps: Step S71, calculating position deviation and posture deviation, namely calculating the position deviation and posture deviation based on the three-dimensional positions and postures of the lock column and the lock hole respectively, and judging corresponding threshold values; Step S72, generating and executing an adaptive control strategy, namely generating a motion control instruction of the gantry crane or the lifting appliance according to the calculated position deviation and attitude deviation, wherein the motion control instruction comprises a X, Y, Z triaxial moving distance and a X, Y, Z triaxial rotating angle; in the moving process of the gantry crane, repeating the steps S3-S7, updating the positions of the lock column and the lock hole in real time, dynamically adjusting the control instruction, and adjusting the moving speed of the gantry crane or the lifting appliance by adopting a speed self-adaptive strategy; if the detection fails or the position deviation exceeds a reasonable range, triggering early warning, suspending automatic control, and waiting for manual intervention or re-detection.
10. The accurate alignment system for the gantry crane lock cylinder lock hole based on the point cloud and the visual depth fusion is used for implementing the accurate alignment method for the gantry crane lock cylinder lock hole based on the point cloud and the visual depth fusion according to any one of claims 1-9, and is characterized by comprising an RGB camera, a laser radar, a GPU computing platform and a PLC control platform, wherein the RGB camera and the laser radar are deployed at a proper position of a gantry crane and are provided with wide-angle lenses, the GPU computing platform is configured to receive two-dimensional images and three-dimensional point cloud data transmitted by the RGB camera and the laser radar, a data preprocessing module, a multi-source data fusion module, a YOLOv target detection module, a point cloud processing module and a deviation computing module are carried, the PLC control platform is connected with the GPU computing platform through signals, is used for receiving deviation computing results, generating X, Y, Z triaxial movement and rotation control instructions of the gantry crane or a lifting tool, and driving an executing mechanism to complete alignment adjustment, supports closed-loop feedback control, receives deviation data updated by the GPU computing platform in real time, dynamically adjusts the control instructions until an alignment threshold is met, and the accurate alignment system has a speed self-adaption strategy and an abnormal warning function.

Description

Portal crane locking cylinder locking hole accurate alignment method and system based on point cloud and visual depth fusion Technical Field The invention relates to the technical field of intelligent port automation, in particular to a method and a system for accurately aligning a lock hole of a gantry crane lock based on point cloud and visual depth fusion. Background The global trade and the China import and export business growth promote the port operation to be automatically and intelligently transformed, the gantry crane is used as core transfer equipment, and the accurate alignment of a lock column and a lock hole is one of key links affecting the operation safety and efficiency. However, the existing gantry crane alignment scheme (manual visual or single sensor) has a plurality of defects: (1) The single vision system is easy to lose efficacy under severe illumination and has large three-dimensional positioning error, and particularly for targets with darker colors such as lockholes, the detection success rate is obviously reduced under a low-illumination environment (< 100 lux) or a strong-light environment (> 10000 lux); (2) The single laser radar system is difficult to achieve accuracy and real-time performance for identifying and positioning small-size targets, and has the problems of huge data volume, sparse point cloud, lack of texture and color information and the like; (3) The traditional calibration method is tedious, time-consuming and high in maintenance cost; (4) The robustness is insufficient in severe weather, so that the system reliability and the operation efficiency are reduced, and even the system cannot work normally; (5) The manual alignment efficiency is low, the precision is unstable and has potential safety hazards, and the high-efficiency operation requirement of modern ports is difficult to meet. Therefore, an automatic alignment method for the lock hole of the gantry crane lock column, which can integrate advantages of multiple sensors and maintain high precision and high robustness in a complex environment, is urgently needed, and meanwhile, has a convenient calibration flow and real-time performance so as to meet the actual application requirements of an intelligent port. Disclosure of Invention The invention aims to provide a method and a system for accurately aligning a lock hole of a gantry crane lock based on point cloud and visual depth fusion, and high-precision, high-efficiency and high-reliability automatic alignment is realized through a full-flow closed-loop design of the point cloud and the visual fusion. The technical scheme adopted by the invention is as follows: A method for accurately aligning a lock cylinder lock hole of a gantry crane based on point cloud and visual depth fusion comprises the following steps: step S1, performing off-line calibration of an RGB camera, namely performing internal reference calibration and distortion correction on the RGB camera based on a standard checkerboard calibration plate to obtain a camera internal reference matrix and a distortion coefficient vector; s2, performing off-line joint calibration on the laser radar and the RGB camera based on a standard checkerboard calibration plate to obtain an external parameter transformation matrix; step S3, the two-dimensional image and the three-dimensional point cloud data are fused, wherein in the operation process of the gantry crane, an RGB camera and a laser radar synchronously acquire the two-dimensional image and the three-dimensional point cloud data of a container operation area, distortion correction is carried out on the two-dimensional image based on a camera internal reference matrix and an external reference transformation matrix, and space-time alignment and fusion are carried out on the two-dimensional image and the three-dimensional point cloud data, so that a distortion correction image and fused three-dimensional point cloud data are obtained; S4, carrying out lock column and lock hole visual segmentation recognition based on YOLOv model, namely adopting a trained YOLOv model to recognize and position lock column and lock hole positions of the container from the distortion correction image in real time; Step S5, projecting a two-dimensional detection result to a three-dimensional point cloud space, namely, based on the detected positions of the lock posts and the lock holes, combining three-dimensional cloud data of fusion points, projecting the two-dimensional detection result to the three-dimensional space to obtain a preliminary projection point cloud, wherein the preliminary projection point cloud comprises a lock post point cloud set and a lock hole point cloud set; s6, carrying out lock cylinder and lock hole positioning based on clustering and geometric fitting, namely extracting three-dimensional positions and postures of lock cylinders and lock holes from the primary projection point cloud by adopting a point cloud clustering and geometric fitting algori