Search

CN-122023499-A - Open pit truck load volume estimation method based on monocular depth and geometric constraint

CN122023499ACN 122023499 ACN122023499 ACN 122023499ACN-122023499-A

Abstract

The invention discloses a monocular depth and geometry constrained strip mine card load volume estimation method which comprises the following steps of 1, obtaining internal and external parameters of a monocular industrial camera and geometric dimension and coordinate system definition of a carriage of a transport truck, 2, detecting an image of a vehicle and the carriage to obtain an image of a region of interest of the carriage, 3, obtaining a pixel level distribution mask of an inner cavity region, a material region and an outer background region of the carriage, extracting an upper edge contour of the carriage, positioning pixel coordinates of four upper edge angular points or preset geometric anchor points of the carriage on the contour, 4, obtaining a measurement depth field under the coordinate system of the carriage, 5, obtaining the volume of a loaded material, and 6, and outputting a load detection result and state information. The method realizes low-cost and high-robustness estimation of the load volume of the strip mine truck, and has the purposes of simple system structure, flexible deployment and strong capability of adapting to complex working conditions.

Inventors

  • FENG ZHIDONG
  • LUO XIAOCHUN
  • ZHANG HUI
  • Tian Jiaqian
  • SONG GUIJUN
  • GUO HONGBO

Assignees

  • 榆林学院

Dates

Publication Date
20260512
Application Date
20260108

Claims (10)

  1. 1. A method for estimating the load volume of a strip mine truck with monocular depth and geometric constraint is characterized by comprising the following steps of; step 1, obtaining internal and external parameters of a monocular industrial camera and defining the geometric dimension and a coordinate system of a carriage of a transport truck, and providing a foundation for subsequent depth scale recovery and three-dimensional reconstruction; Step 2, acquiring an image containing a carriage of a transport truck through a fixed-mounted monocular industrial camera, and detecting the image of the vehicle and the carriage to obtain an image of a region of interest of the carriage; Step 3, inputting an image of a carriage collected by a camera as a target image into a semantic segmentation network to obtain a pixel level segmentation mask of an inner cavity area, a material area and an outer background area of the carriage; step 4, inputting the carriage region of interest image into a monocular depth estimation network to obtain a relative depth map, combining a semantic segmentation result and carriage upper edge key points, and completing camera attitude solving and depth scale recovery under the geometric constraint of the carriage to obtain a measurement depth field under a carriage coordinate system; Step 5, back-projecting the measured depth field into a three-dimensional point cloud in a carriage coordinate system, applying the compartment inner cavity feasible region constraint to the three-dimensional point cloud, fitting the bottom surface of the carriage, and integrating the material height field by adopting a voxelized method to obtain the volume of the loaded material; and 6, inquiring the density of the coal materials in the material safety data table, converting the volume of the loaded materials into loading quality, setting underload, normal and overload judging thresholds by combining the rated loading quality of the vehicle, and outputting loading capacity detection results and state information.
  2. 2. The method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 1, wherein the step 1 is specifically: step 1.1, arranging monocular industrial cameras above or beside a loading point or a unloading point of an open pit mine, and enabling the top view or the side view of the cameras to cover a parking area of a transport truck so as to ensure that a carriage is basically in a field of view of the cameras when the carriage is at a parking position; step 1.2, calibrating a monocular industrial camera by using a calibration plate in an experimental field to obtain a camera internal reference matrix: , Wherein, the , Is focal length [ (] ) Is the principal point coordinates; Step 1.3, defining a geometric model of a carriage inner cavity under a carriage coordinate system according to the technical data or the actual measurement data of the vehicle, and recording the carriage coordinate system as The origin is positioned at the geometric center of the bottom surface of the carriage, The axle is oriented in the forward direction of the vehicle, The axle is arranged in the transverse direction of the vehicle, The axis is vertical upwards, and the length, width and height of carriage are respectively And three-dimensional coordinates of four corner points and other geometric anchor points of the upper edge of the carriage are recorded 。
  3. 3. The method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 2, wherein the step 2 is specifically: step 2.1, acquiring an image sequence containing a strip mine transportation truck carriage through a fixedly installed monocular industrial camera, and recording the original image as follows: Wherein, the For the pixel coordinates, The number of frames is acquired; Step 2.2, utilizing a target detection algorithm based on deep learning to acquire an original image Processing, automatically identifying the objects of the transport truck and the carriage in the image, and outputting a two-dimensional bounding box of the carriage in the image The surrounding frame The method is used for representing the spatial position range of the carriage in the current image and is used as a basis for subsequent image clipping and carriage area analysis. Based on the obtained carriage surrounding frame The original image is subjected to region clipping, and an interesting region image only comprising a carriage region is extracted, wherein the expression is as follows: : providing input for subsequent processing.
  4. 4. The method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 3, wherein in the step 2.1, the acquisition is triggered, and after the transportation truck enters the loading station and stops, the transportation truck passes through the ground induction coil, Or the dispatching system signal triggers the monocular industrial camera to start image acquisition to obtain one or more frames of original images The multi-source trigger signal is combined with the loading station scene, and the image acquisition is triggered only when the transport truck completely enters the loading station and is in a stable parking state.
  5. 5. The method for estimating the load volume of the strip mine truck with monocular depth and geometric constraint according to claim 3, wherein the step 3 is specifically: Step 3.1, imaging the region of interest of the carriage Inputting a pre-trained semantic segmentation network to obtain a pixel classification result: thereby obtaining the pixel set in the carriage cavity Material pixel set Background collection ; Step 3.2, in Boundary-wise extraction of (2) contour line of upper edge of carriage ; Step3.3, contour the upper edge of the carriage Pixel coordinates of key points of the upper edge of the upper detection carriage: Wherein, the And the four upper edge corner points respectively correspond to the front left, the front right, the rear left and the rear right of the carriage.
  6. 6. The method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 5, wherein the step4 is specifically: Step 4.1, the Inputting a monocular depth estimation network to obtain a relative depth map: wherein, the relative depth value is only in the carriage area Has significance; Step 4.2, performing mask constraint by using a semantic segmentation result to construct a carriage inner cavity relative depth field: step 4.3, inputting the carriage region of interest image into a monocular depth estimation network to obtain a relative depth map The depth values of the carriage inner cavity and the material area are reserved only by utilizing the segmentation mask in the step 3, so that a relative depth field is formed Repairing depth holes and peaks by neighborhood filtering to obtain smooth relative depth field ; Step 4.4, solving the camera pose and recovering the scale, and utilizing the three-dimensional coordinates of the key points of the upper edge of the carriage Coordinate with corresponding pixel Matrix of parameters within known cameras Under the condition of (1), solving a rotation matrix of a camera relative to a carriage coordinate system through a PnP algorithm Translation vector The method comprises the following steps: Wherein, the In order to rotate the matrix is rotated, In order to translate the vector of the vector, Solving R and t by PnP algorithm to obtain external parameters of the camera relative to a carriage coordinate system; Hypothesis metric depth And relative depth of The affine relationship is satisfied: Wherein, the As a scale factor of the dimensions of the device, Is the offset; Estimating scale factors using least squares methods by comparing theoretical depths given by geometric models to relative depth values at several control points And offset amount : Obtaining And Converting the relative depth field into a metric depth field accordingly 。
  7. 7. The method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 6, wherein the step 5 is specifically: Step 5.1, back projecting each pixel of the carriage inner cavity area to a carriage coordinate system by adopting the measurement depth field and the camera inner and outer parameters to obtain three-dimensional point cloud data representing the carriage and the material surface; Step 5.2, establishing a carriage bottom plane equation, side plates and a tail gate plane boundary according to a carriage geometric model, performing feasible region cutting on point clouds, deleting points positioned on the outer sides of the side plates, projecting points positioned below the bottom surface to the vicinity of the bottom surface, separating rib plate area point clouds according to the effective volume of a carriage inner cavity, and fitting the carriage bottom surface through a RANSAC algorithm so as to reduce the influence caused by vehicle attitude change and installation errors; And 5.3, dividing the inner cavity space of the carriage into regular voxel grids according to a preset resolution under a carriage coordinate system, mapping the effective point cloud to voxel units, counting the material heights of all horizontal grid columns, performing discrete integration, and deducting invalid volumes such as rib plates, fillets and the like by combining a carriage structure model to obtain a total volume estimated value of the loaded materials.
  8. 8. The method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 7, wherein the step 5.1 is specifically: step 5.1.1, according to the measured depth field and the parameters inside and outside the camera, for each pixel in the carriage In the camera coordinate system The lower three-dimensional coordinate point is : Wherein, the Is the horizontal and vertical pixel coordinates in the image coordinate system, Is a measure of the depth field, Including the focal length and principal point coordinates, Is the pixel coordinates in the image coordinate system; Step 5.1.2 by external reference Transform it into carriage coordinate system Three-dimensional coordinate point converted into carriage coordinate system : Wherein R is the rotation matrix from the camera coordinate system To carriage coordinate system T is a translation from the camera coordinate system to the car coordinate system; Obtaining a three-dimensional point cloud set of the inner cavity of the carriage and the surface of the material : ; The step 5.2 specifically comprises the following steps: step 5.2.1, according to the geometric model of the carriage, setting a carriage bottom plane equation as follows: The plane equation of the side plate and the tail gate is The feasible region is defined as: Only remain to satisfy Constitutes a constrained point cloud set Removing abnormal points outside the carriage; Step 5.2.2, further utilize Algorithm Middle fitting bottom plane The carriage bottom surface reference plane is obtained through the fitting, and the mathematical model of the plane fitting is as follows: Wherein, the As a planar normal vector component, Is a plane constant term; Sample points are iteratively selected through a RANSAC algorithm, and effective points are obtained Performing plane fitting, selecting an optimal model, and finally solving plane parameters of the bottom surface of the carriage ; The step 5.3 specifically comprises the following steps: Step 5.3.1, in the carriage coordinate system, along Direction in step size Dividing a voxel grid: Wherein, the The effective length, width and height of the carriage and the voxel resolution are determined; Step 5.3.2, point cloud The points in (a) are mapped to a voxel grid Statistics of each plane Maximum height above: ; At the height of the bottom surface The height of the material in the columnar area is as follows: Step 5.3.3, integrating the volumes of all voxels to obtain the total volume of the material in the effective space of the inner cavity of the carriage, deducting the contributions of the ineffective volumes of the rib plates and the fillets according to the carriage structure model to obtain the volume estimated value of the loaded material, and then loading the volume Approximated as a discrete integral over a two-dimensional lattice point: 。
  9. 9. the method for estimating the loading volume of the strip mine truck with monocular depth and geometric constraint according to claim 8, wherein the step 6 is specifically: step 6.1, estimating the loading quality according to the material density provided by the mine management system Or the statistical density obtained by checking the wagon balance data, and converting the loading volume into the loading mass: ; step 6.2, loading state judgment and result output, namely setting the rated loading mass of the vehicle as The underload and overload thresholds are respectively And Wherein the method comprises the steps of Loading state The judgment is as follows: Will be Loading state Output to the result display and alarm terminal, when Or under-load, an alarm is triggered.
  10. 10. A load detection system for implementing the method of any one of claims 1-9, comprising monocular industrial cameras, a fixed support, an edge calculation unit, a mine dispatching system, and a result display and alarm terminal in communication with the mine dispatching system; The monocular industrial camera is fixedly arranged at the beam above the loading point and is used for collecting the carriage and material images of the strip mine transport truck in the state of stopping at the loading station; The fixed bracket is used for supporting and positioning the monocular industrial camera so as to ensure that the visual angle of the camera is stable and cover the effective monitoring area of the loading station; The edge computing unit is in communication connection with the monocular industrial camera and is used for receiving image data acquired by the monocular industrial camera, locally executing processing procedures such as target detection, depth estimation, load volume computation and the like, and generating a load capacity detection result; The mine dispatching system is in communication connection with the edge computing unit and is used for receiving the loading capacity detection result and correlating the result with mine production dispatching data so as to support transportation dispatching and production management; the result display is in communication connection with the alarm terminal and the mine dispatching system, and is used for visually displaying the load detection result and outputting alarm information when the detection result does not meet the preset loading condition.

Description

Open pit truck load volume estimation method based on monocular depth and geometric constraint Technical Field The invention relates to the technical field of mine intellectualization and computer vision, in particular to a strip mine card load volume estimation method based on monocular depth and geometric constraint. Background In surface mine production, whether the ore card loading reaches the planned amount directly influences scheduling efficiency and cost accounting. The traditional load volume measurement is dependent on wagon balance [ CN121140920A ], laser scanning [10.19650/j.cnki.cjsi.J2311797] or binocular camera [10.27393/d.cnki.gxazu.2024.000260], and has the problems of high equipment cost, complex construction and transformation, complex maintenance, limited installation and the like. The monocular depth estimation is rapidly developed in recent years, however, the direct calculation of volume errors is larger due to scale uncertainty and geometric mismatch, and meanwhile, factors such as dust, strong backlight, geometrically complex materials, carriage boundary shielding and the like in a mining area scene are challenging to semantic segmentation and depth estimation. Therefore, there is an urgent need for a volume estimation method that introduces a geometric prior and a depth scale constraint of a carriage under a monocular condition, and improves accuracy and stability of voxel reconstruction and volume integration by pertinently improving a semantic segmentation network and depth output optimization. Disclosure of Invention In order to overcome the technical problems, the invention aims to provide a strip mine truck load volume estimation method with monocular depth and geometric constraint, which solves the problems of high cost, complex deployment, difficult maintenance, insufficient load capacity detection precision and robustness under complex working conditions such as strip mine strong light, dust and the like in the existing means such as wagon balance weighing, laser radar or binocular vision, thereby realizing low-cost and high-robustness estimation on the strip mine truck load volume and having the purposes of simple system structure, flexible deployment and strong capability of adapting to complex working conditions. In order to achieve the above purpose, the technical scheme adopted by the invention is as follows: A method for estimating the load volume of a strip mine truck with monocular depth and geometric constraint comprises the following steps of; step 1, camera installation calibration and carriage geometric modeling: obtaining the internal and external parameters of a monocular industrial camera and the definition of the geometric dimension and the coordinate system of a carriage of a transport truck, and providing a foundation for the subsequent depth scale recovery and three-dimensional reconstruction; Step 2, image acquisition and carriage area extraction: acquiring an image containing a carriage of a transport truck through a fixed-mounted monocular industrial camera, and detecting the image of a vehicle and the carriage to obtain an image of a region of interest of the carriage; step 3, compartment semantic segmentation and upper edge key point extraction: The method comprises the steps of inputting images of a carriage collected by a camera into a semantic segmentation network as target images to obtain pixel level segmentation masks of an inner cavity area, a material area and an outer background area of the carriage, extracting upper edge contours of the carriage, positioning pixel coordinates of four upper edge corner points or preset geometric anchor points of the carriage on the contours, and solving subsequent geometric constraints; Step 4, the monocular depth estimation and the scale recovery under the geometric constraint of the carriage: Inputting the carriage interest region image into a monocular depth estimation network to obtain a relative depth map, combining a semantic segmentation result and carriage upper edge key points, and completing camera attitude solving and depth scale recovery under the geometric constraint of the carriage to obtain a measurement depth field under a carriage coordinate system; Step 5, three-dimensional point cloud reconstruction and voxel volume calculation: back-projecting the measured depth field into a three-dimensional point cloud in a carriage coordinate system, applying the space constraint of the carriage inner cavity to the three-dimensional point cloud, fitting the carriage bottom surface, and integrating the material height field by adopting a voxelization method to obtain the volume of the loaded material; Step 6, outputting a loading amount result and judging the state: And inquiring the density of the coal materials in the material safety data table, converting the volume of the loaded materials into loading quality, setting underload, normal and overload judging thresholds by combining the rated loading q