CN-115439322-B - Vehicle-mounted looking-around splicing method and system based on deformable registration

CN115439322BCN 115439322 BCN115439322 BCN 115439322BCN-115439322-B

Abstract

The invention relates to a vehicle-mounted look-around splicing method and system based on deformable registration, wherein the method comprises the following steps of S1, acquiring a non-planar view by a four-way fisheye vehicle-mounted camera, correcting the non-planar view to obtain a planar view, S2, optimizing the planar view through minimum parallax tolerance depth to obtain an optimized planar view, using an ORB algorithm to obtain a global alignment transformation matrix H, S3, representing the optimized planar view by grids, repositioning corresponding grid vertices fitted by the optimized planar view by utilizing H to align matching points, defining a constraint energy function to inhibit deformation of grids to obtain the planar view after grid registration, and S4, fusing the planar view after grid registration by a linear weighting method and a gain compensation method to obtain the spliced four-way fisheye vehicle-mounted ring view. According to the method provided by the invention, panoramic view and look-around splicing of the four-way fisheye camera can be realized without depending on camera parameters, and the image fusion method is utilized to effectively optimize the artifact.

Inventors

LING QIANG
XIONG JIABING

Assignees

中国科学技术大学

Dates

Publication Date: 20260505
Application Date: 20220811

Claims (4)

1. The vehicle-mounted looking-around splicing method based on deformable registration is characterized by comprising the following steps of: step S1, acquiring a non-planar view by a four-way fisheye vehicle-mounted camera, and correcting the non-planar view to obtain a planar view; Step S2, optimizing the plane view through the minimum parallax tolerance depth to obtain an optimized plane view, and obtaining a global alignment transformation matrix by using an ORB algorithm The method specifically comprises the following steps: step S21 of depth by increasing minimum parallax tolerance for the planar view Optimizing to obtain an optimized plane view, as shown in a formula (6): (6) Wherein, the For the radius of the lens of the camera, Representing the field of view of each fisheye camera lens ; Respectively representing the lens vertex and the object position point, F representing the lens far-viewing point, Representing a minimum visible depth of the camera; step S22, order And Representing two adjacent said optimized plan views, computing using ORB algorithm And Matching points among the two, and then obtaining a global alignment transformation matrix through calculation Step S3, representing the optimized plane view by using grids, and utilizing Repositioning corresponding grid vertexes of the optimized plane view fitting to align matching points, and restraining deformation of the grid by defining constraint energy functions, wherein the constraint energy functions comprise constraint energy functions of feature alignment, constraint energy functions of local structure maintenance and constraint energy functions of global similarity, and the method comprises the following steps of: Step S31, representing the optimized plane view by a grid, and utilizing the grid deformation For a pair of And Repositioning corresponding grid vertexes of the middle overlapping area, so that grid alignment is realized; Defining a constraint energy function to suppress mesh deformation, defining a feature alignment constraint energy function, as shown in equation (7): (7) Wherein, the From images And an image All of the distances between Matching the vertices; Representation and image All adjacent images with overlapping areas; And Representing weighting factors, mesh vertices deformed by the mesh Or (b) In the target image Or (b) Position determination in the grid cells before deformation; Solving a least square solution of the formula (7), and ensuring that the distance between each pair of matching points in the overlapped image area is as short as possible; Step S32, defining a constraint energy function of local structure maintenance, as shown in a formula (8): (8) Wherein, the Is an image The set of each opposite side of the grid before and after middle deformation, T is a similar transformation matrix; representing an image The edge of the grid before deformation on the upper surface, Respectively representing images Up-deformed grid edge, n is image The number of middle grid edges; step S33, defining an energy function of global similarity constraint, wherein the energy function is shown in formulas (9) - (10): (9) (10) Wherein, the For the scale factor of each source image, Is a twiddle factor; And Is the component of T in the x-axis and y-axis; Is a weighting coefficient; and S4, fusing the planar views subjected to grid registration by adopting a linear weighting method and a gain compensation method to obtain a spliced four-way fisheye vehicle-mounted annular view.
2. The vehicle-mounted look-around stitching method based on deformable registration according to claim 1, wherein the step S1 is to obtain a non-planar view by a four-way fisheye vehicle-mounted camera, correct the non-planar view to obtain a planar view, and specifically includes: step S11, cutting the non-planar view to an edge length L, wherein each image point of the non-planar view As spherical coordinates Mapping to a 3D unit sphere, wherein, Is the azimuth; is the elevation angle; step S12, calculating Cartesian coordinates of (c) As shown in formulas (1) - (3): (1) (2) (3) Wherein, the The pitch angle between the center line of the fish-eye lens and the horizontal plane of the camera can be self-calibrated or obtained from a camera provider; step S13, spherical projection is used Point to Point Projecting to a 2D orthogonal coordinate system, wherein the 2D orthogonal coordinate system is shown in formulas (4) - (5): (4) (5) Wherein, the Representing the field of view of each camera lens, L is the length of the non-planar view.
3. The vehicle-mounted view-looking-around splicing method based on deformable registration according to claim 2, wherein the step S4 is to fuse the planar views after grid registration by a linear weighting method and a gain compensation method to obtain a spliced four-way fisheye-mounted view-around, and specifically comprises the following steps: Step S41, the linear weighting method utilizes a ramp function to fuse the overlapped area of the two images, wherein the size is that R rows and c columns of blended pixels of overlapping region of (c) As shown in formula (11): (11) Wherein, the And Is the position Pixels taken from the left image and the right image, respectively; when the right-hand overlap region is blended, , , Is the width of the mixing zone; When the left-hand overlap region is blended, , ; Step S42, reducing color or brightness difference between images shot by different camera lenses by using a gain compensation method, wherein the images And (3) with Compensation factor between As shown in formula (12): (12) Wherein, the Representing an image And image The amount of pixels in the overlap region; And Respectively representing images And image Average pixel intensity of the overlapping region; And The gain and standard deviation of pixel intensity errors, respectively.
4. A vehicle-mounted look-around stitching system based on deformable registration, comprising the following modules: the plane view acquisition module is used for acquiring a non-plane view by the four-way fisheye vehicle-mounted camera, correcting the non-plane view and obtaining a plane view; The global alignment transformation matrix acquisition module is used for optimizing the plane view through the minimum parallax tolerance depth to obtain an optimized plane view, and obtaining a global alignment transformation matrix by using an ORB algorithm The method specifically comprises the following steps: step S21 of depth by increasing minimum parallax tolerance for the planar view Optimizing to obtain an optimized plane view, as shown in a formula (6): (6) Wherein, the For the radius of the lens of the camera, Representing the field of view of each fisheye camera lens ; Respectively representing the lens vertex and the object position point, F representing the lens far-viewing point, Representing a minimum visible depth of the camera; step S22, order And Representing two adjacent said optimized plan views, computing using ORB algorithm And Matching points among the two, and then obtaining a global alignment transformation matrix through calculation ; Grid registration plane view module for representing the optimized plane view with grid, using Repositioning corresponding grid vertexes of the optimized plane view fitting to align matching points, and restraining deformation of the grid by defining constraint energy functions, wherein the constraint energy functions comprise constraint energy functions of feature alignment, constraint energy functions of local structure maintenance and constraint energy functions of global similarity, and the method comprises the following steps of: Step S31, representing the optimized plane view by a grid, and utilizing the grid deformation For a pair of And Repositioning corresponding grid vertexes of the middle overlapping area, so that grid alignment is realized; Defining a constraint energy function to suppress mesh deformation, defining a feature alignment constraint energy function, as shown in equation (7): (7) Wherein, the From images And an image All of the distances between Matching the vertices; Representation and image All adjacent images with overlapping areas; And Representing weighting factors, mesh vertices deformed by the mesh Or (b) In the target image Or (b) Position determination in the grid cells before deformation; Solving a least square solution of the formula (7), and ensuring that the distance between each pair of matching points in the overlapped image area is as short as possible; Step S32, defining a constraint energy function of local structure maintenance, as shown in a formula (8): (8) Wherein, the Is an image The set of each opposite side of the grid before and after middle deformation, T is a similar transformation matrix; representing an image The edge of the grid before deformation on the upper surface, Respectively representing images Up-deformed grid edge, n is image The number of middle grid edges; step S33, defining an energy function of global similarity constraint, wherein the energy function is shown in formulas (9) - (10): (9) (10) Wherein, the For the scale factor of each source image, Is a twiddle factor; And Is the component of T in the x-axis and y-axis; Is a weighting coefficient; And the plane view splicing module is used for fusing the plane views subjected to grid registration by adopting a linear weighting method and a gain compensation method to obtain a spliced four-way fisheye vehicle-mounted annular view.

Description

Vehicle-mounted looking-around splicing method and system based on deformable registration Technical Field The invention relates to the field of computer vision and automobile automatic driving, in particular to a vehicle-mounted look-around splicing method and system based on deformable registration. Background The on-board panoramic view in the form of a bird's eye view, which provides the omni-directional visual perception of the surroundings of the vehicle, is a critical and challenging task in automatic driving. Compared with a high-cost laser radar solution, the use of the cheap fisheye camera has a great competitive advantage in cost, but the problems of deformation, irregular boundary, calculation cost and the like which can occur when the fisheye camera is spliced in a non-planar view correspondingly have higher requirements on the performance of the look-around splicing algorithm. With fisheye cameras mounted to four orthogonal viewing directions of a vehicle and each having a field-of-view of greater than 180 deg. while simultaneously capturing images having a large proportion of overlapping areas that collectively cover the entire 360 deg. viewing range of the surrounding environment, the images can be stitched into a panoramic image in the form of a bird's eye view using a look-around stitching algorithm. Most current look-around stitching methods use rigid image alignment based on geometric transformations, which requires hardware parameter information such as lens aperture, focal length, assembly, etc. of a four-way fisheye camera, but these information are typically proprietary to the camera vendor or require additional measurement procedures. These look-around stitching methods, which rely on specific camera-related parameters and limited camera selection and closed commercial software, limit the general purpose vehicle panoramic program development. Therefore, how to realize the look-around stitching of four-way fisheye images without depending on camera parameters, and at the same time, to alleviate inconsistent artifacts caused by inaccurate pixel alignment and camera luminosity difference becomes a problem to be solved. Disclosure of Invention In order to solve the technical problems, the invention provides a vehicle-mounted looking-around splicing method and system based on deformable registration. The technical scheme of the invention is that the vehicle-mounted looking-around splicing method based on deformable registration comprises the following steps: step S1, acquiring a non-planar view by a four-way fisheye vehicle-mounted camera, and correcting the non-planar view to obtain a planar view; step S2, optimizing the plane view through the minimum parallax tolerance depth to obtain an optimized plane view, and obtaining a global alignment transformation matrix H by using an ORB algorithm; S3, representing the optimized plane view by using a grid, repositioning corresponding grid vertexes fitted by the optimized plane view by using H to align the matched points, and restraining deformation of the grid by defining constraint energy functions, wherein the constraint energy functions comprise constraint energy functions of characteristic alignment, constraint energy functions of local structure maintenance and constraint energy functions of global similarity; and S4, fusing the planar views subjected to grid registration by adopting a linear weighting method and a gain compensation method to obtain a spliced four-way fisheye vehicle-mounted annular view. Compared with the prior art, the invention has the following advantages: 1. the existing fisheye camera splicing method needs to measure a series of hardware parameters of the four-way fisheye camera, such as lens aperture, focal length, assembly error and the like. But typically these parameters are either proprietary to the camera vendor or require additional closed business software for measurement calibration, which can impose limitations on limited camera models, etc. in terms of more abundant scene popularization for fisheye camera panoramic view stitching. The invention discloses a vehicle-mounted looking-around splicing method based on deformable registration, which can realize panoramic looking-around splicing of a four-way fisheye camera under the condition of not depending on camera parameters, is more suitable for developing application programs in a general scene, and brings better popularization prospect for the technology. 2. Because the four-way cameras are different in installation positions, the optical centers of the four-way cameras are not aligned to cause parallax of 4-way images, so that artifacts appear in a merging area, the four-way images are independently installed and processed, the luminance characteristics of the four-way images are often inconsistent due to different illumination conditions, and finally, the corresponding points of overlapping areas of the images shot by adjacent cameras (such as front an