CN-122023118-A - Feature constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method

CN122023118ACN 122023118 ACN122023118 ACN 122023118ACN-122023118-A

Abstract

The invention discloses a characteristic constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method, and relates to the technical field of underwater optical measurement and three-dimensional reconstruction. The method comprises the steps of firstly calibrating camera parameters and refraction interface parameters of an underwater binocular vision system, carrying out enhancement processing on acquired multi-sequence underwater images, extracting two-dimensional characteristic points of adjacent frames, constructing direction distribution constraint screening reliable matching point pairs, completing image splicing, carrying out three-dimensional reconstruction and refraction error correction on each station image pair by utilizing a multi-refraction plane correction model to obtain local point clouds, mapping the two-dimensional characteristic points to a three-dimensional space to generate the three-dimensional characteristic point pairs, and calculating a rigid body transformation matrix between the adjacent point clouds by adopting a double-time exchange ICP algorithm combined with the constraint of the three-dimensional characteristic points to realize high-precision splicing of the multi-sequence point clouds. The invention effectively solves the problems of poor underwater imaging quality, refractive distortion and splice accumulated error by fusing two-dimensional-three-dimensional multi-level characteristic constraint and accurate refraction correction.

Inventors

HOU SHITONG
WU TAO
SUN WEIHAO
SHEN HAN
XIONG WEN
WU ZHISHEN
HE XIAOYUAN

Assignees

东南大学

Dates

Publication Date: 20260512
Application Date: 20260130

Claims (10)

1. A characteristic constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method is characterized by comprising the following steps: S1, receiving calibration parameters of an underwater binocular vision system camera and a multi-sequence original image of an underwater scene, wherein the calibration parameters comprise an internal reference matrix, a distortion coefficient, a binocular relative pose and a refraction interface parameter; S2, carrying out enhancement processing on the original underwater image sequence received in the step S1 to obtain an image sequence with enhanced characteristics; S3, extracting two-dimensional feature points of adjacent frame images in the image sequence by adopting a feature recognition and matching algorithm, and matching to obtain a two-dimensional matching feature point pair set and pixel coordinates (u, v) of the two-dimensional matching feature point pair set in each image; S4, constructing characteristic point distribution constraint conditions in the vertical direction and the horizontal direction based on the two-dimensional matching characteristic point pairs obtained in the step S3, screening reliable matching point pairs conforming to space continuity constraint, calculating a homography transformation matrix or perspective transformation matrix among images based on the reliable matching point pairs, and sequentially completing splicing of multiple sequences of two-dimensional images to form a two-dimensional panoramic image; S5, constructing a multi-refraction plane correction model by using the calibration parameters and the refraction parameters acquired in the step S1 aiming at the original left-right view image pair of each measuring station acquired in the step S1, carrying out three-dimensional reconstruction on the left-right view of each measuring station in the multi-sequence original image, and carrying out refraction error correction on the reconstructed three-dimensional point coordinates to acquire a three-dimensional point cloud under a local coordinate system of each measuring station; s6, mapping the two-dimensional image feature point pairs obtained in the step S3 to the three-dimensional point cloud according to the camera perspective projection model and the three-dimensional point cloud obtained in the step S5, generating three-dimensional feature point pairs, and constructing direction feature constraint conditions in a three-dimensional space based on the three-dimensional feature point pairs; And S7, calculating a rigid body transformation matrix between three-dimensional point clouds of two adjacent measuring stations by adopting a double-time exchange ICP algorithm based on the constraint condition of the three-dimensional characteristic point pairs, and unifying all local point clouds under a global coordinate system by sequentially multiplying the transformation matrix between the adjacent measuring stations so as to realize accurate splicing of the multi-sequence underwater three-dimensional point clouds.
2. The method for splicing the multi-sequence underwater image and the three-dimensional point cloud based on the characteristic constraint of claim 1, wherein the underwater binocular vision system comprises two cameras which are fixed in a waterproof shell, the base line length of the cameras is fixed, the cameras are provided with light supplementing lamps, the original underwater image sequence is obtained by carrying the underwater binocular vision system on an underwater robot, and multi-station continuous image acquisition is carried out on an underwater structure or scene; the underwater calibration adopts a customized underwater calibration plate, and key refraction parameters such as normal vector n and distance d of the waterproof glass interface relative to a camera are obtained through calculation by analyzing distortion differences of images of the underwater calibration plate and images of the air calibration plate.
3. The feature constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method is characterized in that a binocular camera is calibrated in air to obtain camera internal parameters, distortion coefficients and binocular relative pose, wherein the binocular relative pose comprises a rotation matrix R and a translation vector T; And calibrating by adopting a customized calibration plate in an underwater environment, and solving refractive interface parameters, wherein the refractive interface parameters comprise a normal vector n of a waterproof glass interface relative to a camera and a distance parameter d by analyzing distortion differences of images in underwater and air.
4. The method for splicing the multi-sequence underwater image and the three-dimensional point cloud based on the feature constraint according to claim 3, wherein the feature recognition and matching algorithm adopts SIFT, SURF or ORB algorithm, and the matching process adopts nearest neighbor distance ratio or cross verification strategy for primary screening.
5. The method for splicing the multi-sequence underwater image and the three-dimensional point cloud based on the characteristic constraint according to claim 3 is characterized in that in the step S4, characteristic point distribution constraint conditions in the vertical direction and the horizontal direction are constructed, and the characteristic point distribution constraint conditions are specifically as follows: For a pair of successfully matched images I t and I t+1 , respectively extracting M and N characteristic points to form K pairs of matching, and calculating difference vectors of coordinates of all the matching point pairs in the two images; the histogram distribution of the components of the statistical difference vector in the horizontal U axis and the vertical V axis of the image coordinate system is counted; setting a threshold value, reserving matching point pairs with difference components in the main peak area of the histogram, and eliminating outlier matching.
6. The method for splicing the multi-sequence underwater images and the three-dimensional point clouds based on the characteristic constraint of claim 5 is characterized in that in the step S4, the homography transformation matrix or the perspective transformation matrix among the images is calculated based on reliable matching point pairs, the splicing of the multi-sequence two-dimensional images is sequentially completed, and the formed two-dimensional panorama is specifically as follows: S41, aiming at the reliable matching point pair set obtained in the step S3, adopting a direct linear transformation algorithm and combining with RANSAC optimization to solve a homography transformation matrix H between adjacent images, and approximating the situation that the underwater scene has obvious parallax or a non-planar structure by adopting a perspective transformation model; S42, setting the coordinates of a source image point as p i =(x i ,y i ,1) T and the coordinates of a target image corresponding point as p i ′=(x i ′,y i ′,1) T , and defining the transformation relationship by a 3×3 matrix H as follows: solving 8 degree of freedom parameters of H through at least 4 groups of matching point pairs, and eliminating mismatching by using RANSAC to obtain a robust optimal transformation matrix; S43, mapping the current image I k to be spliced to a coordinate system of the reference image I ref according to a transformation matrix H, and calculating pixel values of each pixel position (x ', y') in the mapped image through bilinear interpolation or cubic convolution interpolation so as to keep the image smooth and reduce geometric distortion; s44, adopting a gradual stitching or global optimization strategy, and orderly aligning and fusing subsequent images by taking a previous stitching result as a new reference frame if the gradual stitching is adopted, and carrying out bundling adjustment on transformation matrixes of all image pairs if the global optimization is adopted, so as to minimize the re-projection errors of all matching point pairs and obtain a globally consistent coordinate frame; S45, adopting a weighted average, multi-band fusion or fusion method based on exposure compensation in an overlapping region, eliminating the difference between a splicing gap and brightness, and using a gradual-in gradual-out weight function for pixels in the overlapping region: The weight w is changed linearly or nonlinearly according to the distance between the pixel and the overlapped boundary, I blend is the final pixel value of a certain pixel point of the fused image in the overlapped area, I src is the original pixel value of the corresponding pixel point of the source image in the overlapped area, and I dst is the original pixel value of the corresponding pixel point of the target image in the overlapped area; S46, fusing all the transformed images onto a unified panoramic canvas, automatically cutting or extrapolating and filling blank areas or irregular edge areas, and finally outputting a seamless two-dimensional panoramic image.
7. The method for splicing the multi-sequence underwater image and the three-dimensional point cloud based on the characteristic constraint of claim 6, wherein in the step S5, a multi-refraction plane correction model is constructed, three-dimensional reconstruction is carried out on left and right views of each measuring station in a multi-sequence original image, refraction error correction is carried out on the reconstructed three-dimensional point coordinates, and the three-dimensional point cloud under the local coordinate system of each measuring station is obtained, wherein the method comprises the following steps: s51, performing stereo matching on the corrected distorted left and right images to obtain a parallax image; S52, primarily calculating a three-dimensional coordinate P cam which does not consider refraction according to the binocular stereoscopic vision principle; S53, carrying out refraction correction based on the Snell's law and a ray tracing method, wherein a core formula of a correction model expresses the relation among a real underwater object point P world , a refraction interface parameter and pixel coordinates (u, v) of an image point; For an image point, the back projection light is refracted through a waterproof glass interface, the correction purpose is to find the coordinate enabling the light to finally point to a real object point, and the following constraint is solved through an iterative optimization or geometric analysis method: wherein O is an incident point on a refractive interface, n is an interface normal vector, and mu is a scale factor related to the refractive index; S534, finally obtaining the corrected accurate three-dimensional coordinate P world ՛ based on the solving result.
8. The method for splicing the multi-sequence underwater image and the three-dimensional point cloud based on the feature constraint of claim 6, wherein in the step S6, the two-dimensional image feature point pair obtained in the step S3 is mapped to the three-dimensional point cloud according to the camera perspective projection model and the three-dimensional point cloud obtained in the step S5, the three-dimensional feature point pair is generated, and the direction feature constraint condition in the three-dimensional space is constructed based on the three-dimensional feature point pair, specifically as follows: For a pair of matched characteristic points m t and m t+1 in the images I t and I t+1 , three-dimensional points F t and F t+1 under respective local coordinate systems can be calculated through three-dimensional reconstruction and coordinate transformation of respective measuring stations, so that a pair of known matched three-dimensional characteristic point pairs is formed, and strong constraint is provided for the next point cloud stitching.
9. The feature constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method is characterized in that in the step S7, a double-time exchange ICP algorithm combined with three-dimensional feature point constraint is adopted, and the specific flow is as follows: s71, inputting a former station source point cloud A, a latter station target point cloud B and a matched three-dimensional characteristic point pair set F in the two point clouds obtained in the step S6; s72, the first ICP (A-B) takes the corresponding distance of the three-dimensional feature point pair F as an additional constraint term, and the additional constraint term is fused into an objective function of ICP, wherein the objective function E is as follows: Wherein R and T are rotation components and translation components of the transformation matrix M 1 , a i , b i is a common point pair found by nearest neighbor search, f aj , f bj is a three-dimensional feature point pair with known matching, lambda is a weight coefficient, and the function is optimized to solve the primary transformation matrix M 1 ; S73, transforming the point cloud B to the vicinity of the coordinate system of A by using M 1 , then exchanging roles, taking the transformed B ՛ as a source and the original A as a target, running the constrained ICP of the step S72 again, and solving a modified transformation matrix M 2 ; S74, final transformation, namely combining the two transformations to obtain a final accurate transformation matrix M from A to B final = M 2 M 1 。
10. A feature constraint-based multi-sequence underwater image and three-dimensional point cloud stitching system for implementing the feature constraint-based multi-sequence underwater image and three-dimensional point cloud stitching method as claimed in any one of claims 1 to 9, characterized by comprising: the underwater binocular vision acquisition assembly comprises two underwater cameras, a waterproof shell, a base line fixing frame and a light supplementing device which are rigidly connected, and is used for acquiring an original underwater image sequence; the underwater carrying platform assembly is an ROV or AUV and is used for carrying and controlling the underwater binocular vision acquisition module to carry out multi-station scanning along a preset path; The water surface control and data processing component comprises a water control unit and a calculation server; the water control unit is used for monitoring the state of the underwater platform and receiving image data; The computing server includes: The data receiving module is used for receiving calibration parameters of the underwater binocular vision system camera and multi-sequence original images of the underwater scene, wherein the calibration parameters comprise an internal reference matrix, a distortion coefficient, binocular relative pose and refractive interface parameters; the enhancement processing module is used for carrying out enhancement processing on the received original underwater image sequence to obtain an image sequence with enhanced characteristics; The feature recognition and matching module is used for extracting two-dimensional feature points of adjacent frame images in the image sequence by adopting a feature recognition and matching algorithm and matching the two-dimensional feature points to obtain a two-dimensional matching feature point pair set and pixel coordinates (u, v) of the two-dimensional matching feature point pair set in each image; The two-dimensional image stitching module is used for constructing characteristic point distribution constraint conditions in the vertical direction and the horizontal direction based on the obtained two-dimensional matching characteristic point pairs, screening reliable matching point pairs conforming to space continuity constraint, calculating a homography transformation matrix or perspective transformation matrix among images based on the reliable matching point pairs, and sequentially completing stitching of the multi-sequence two-dimensional images to form a two-dimensional panoramic image; The model construction module is used for constructing a multi-refraction plane correction model by utilizing the acquired calibration parameters and refraction parameters according to the acquired original left-right view image pair of each measuring station, carrying out three-dimensional reconstruction on the left-right view of each measuring station in the multi-sequence original image, and carrying out refraction error correction on the reconstructed three-dimensional point coordinates to obtain a three-dimensional point cloud under a local coordinate system of each measuring station; The mapping module is used for mapping the obtained two-dimensional image characteristic point pairs to the three-dimensional point cloud according to the camera perspective projection model and the obtained three-dimensional point cloud, generating three-dimensional characteristic point pairs and constructing direction characteristic constraint conditions in a three-dimensional space based on the three-dimensional characteristic point pairs; The three-dimensional point cloud splicing module is used for calculating rigid body transformation matrixes between the three-dimensional point clouds of two adjacent measuring stations by adopting a twice-to-ICP algorithm based on constraint conditions of three-dimensional characteristic point pairs, and unifying all local point clouds under a global coordinate system by sequentially accumulating the transformation matrixes between the adjacent measuring stations so as to realize accurate splicing of the multi-sequence underwater three-dimensional point clouds.

Description

Feature constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method Technical Field The invention relates to the technical field of underwater optical measurement and three-dimensional reconstruction, in particular to a multi-sequence underwater image and three-dimensional point cloud splicing method based on characteristic constraint. Background With the increasing demands of ocean resource development, harbor channel construction and operation and maintenance of underwater engineering, periodic detection, disease identification and three-dimensional morphology mapping of underwater structures (such as ship bodies, pipelines, dam bodies and ocean platforms) have become key links for guaranteeing the safe operation of ocean engineering. The underwater three-dimensional reconstruction technology based on optical vision has been widely used in the fields of underwater target detection, structural health monitoring, underwater archaeology and the like by virtue of the advantages of non-contact, high resolution, rich information and the like. However, the conventional vision measurement technology is limited by the complex underwater physical environment and imaging conditions, and is still faced with a plurality of bottlenecks when dealing with high-precision reconstruction tasks of a large-scale scene, namely 1) the underwater imaging quality is poor, the image contrast is low, the characteristics are fuzzy due to light attenuation, scattering and color distortion, and the failure rate of the conventional characteristic matching algorithm is high. 2) Refraction interference, namely, a camera is arranged in a waterproof shell, light passes through glass and water to be refracted, so that systematic errors are generated in three-dimensional reconstruction based on a pinhole model, and the accuracy of subsequent point cloud splicing is directly affected. 3) The splicing accumulated error is large: whether the image is spliced or the point cloud is spliced (such as an ICP algorithm), when the sequence data is longer, the matching errors are accumulated step by step, so that the head and the tail cannot be closed or the model is seriously distorted. Disclosure of Invention The invention aims to provide a characteristic constraint-based multi-sequence underwater image and three-dimensional point cloud splicing method, which realizes high-precision seamless two-dimensional panoramic and three-dimensional model reconstruction of underwater large-range scenes by fusing two-dimensional and three-dimensional characteristic constraints and combining underwater refraction correction and an improved point cloud registration algorithm. According to the first aspect of the invention, in order to achieve the above purpose, the invention provides a multi-sequence underwater image and three-dimensional point cloud splicing method based on characteristic constraint, which comprises the following steps: S1, receiving calibration parameters of an underwater binocular vision system camera and a multi-sequence original image of an underwater scene, wherein the calibration parameters comprise an internal reference matrix, a distortion coefficient, a binocular relative pose and a refraction interface parameter; S2, carrying out enhancement processing on the original underwater image sequence received in the step S1 to obtain an image sequence with enhanced characteristics; S3, extracting two-dimensional feature points of adjacent frame images in the image sequence by adopting a feature recognition and matching algorithm, and matching to obtain a two-dimensional matching feature point pair set and pixel coordinates (u, v) of the two-dimensional matching feature point pair set in each image; S4, constructing characteristic point distribution constraint conditions in the vertical direction and the horizontal direction based on the two-dimensional matching characteristic point pairs obtained in the step S3, screening reliable matching point pairs conforming to space continuity constraint, calculating a homography transformation matrix or perspective transformation matrix among images based on the reliable matching point pairs, and sequentially completing splicing of multiple sequences of two-dimensional images to form a two-dimensional panoramic image; S5, constructing a multi-refraction plane correction model by using the calibration parameters and the refraction parameters acquired in the step S1 aiming at the original left-right view image pair of each measuring station acquired in the step S1, carrying out three-dimensional reconstruction on the left-right view of each measuring station in the multi-sequence original image, and carrying out refraction error correction on the reconstructed three-dimensional point coordinates to acquire a three-dimensional point cloud under a local coordinate system of each measuring station; s6, mapping the two-dimensional image feature point pairs obtained in the