US-12621571-B2 - Camera-to-camera transformation using multiple region matching and bundle adjustment

US12621571B2US 12621571 B2US12621571 B2US 12621571B2US-12621571-B2

Abstract

A vehicle, system and method of aligning a first camera of the vehicle with a second camera of the vehicle. The system includes a processor configured to detect a first set of features in a first image obtained from the first camera, detect a second set of features in a second image obtained from the second camera, determine a first region of interest in the first image, determine a second region of interest in the second image, generate a feature pair by matching the first set of features to the second set of features using the first region of interest and the second region of interest, align the first camera with the second camera using the feature pair, and control an operation at the vehicle based on the alignment between the first camera and the second camera.

Inventors

Binbin Li
Xinyu Du
Yao Hu
Hao Yu

Assignees

GM Global Technology Operations LLC

Dates

Publication Date: 20260505
Application Date: 20230215

Claims (20)

1 . A method of aligning a first camera of a vehicle with a second camera of the vehicle, comprising: detecting a first set of features in a first image obtained from the first camera; detecting a second set of features in a second image obtained from the second camera; determining a first region of interest in the first image; determining a second region of interest in the second image; generating a feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest, wherein minimizing the cost function includes minimizing a descriptor distance for the feature pair, a difference in color of a first pixel in the first image and a second pixel in the second image, the first pixel and the second pixel associated with the feature pair, and a difference in a number of features in a given segmented region of the first region of interest and the second region of interest; aligning the first camera with the second camera using the feature pair; and controlling an operation at the vehicle based on the alignment between the first camera and the second camera.
2 . The method of claim 1 , further comprising determining a transformation matrix between the first camera and the second camera using the feature pairs.
3 . The method of claim 1 , further comprising storing the feature pair in a queue and retrieving the feature pair from the queue using a sliding window.
4 . The method of claim 1 , further comprising defining the first region of interest in the first image and the second region of interest in the second image, the first region of interest corresponding to the second region of interest, detecting the first set of features and determining a first set of segmented regions within the first region of interest, and detecting the second set of features and determining a second set of segmented regions within the second region of interest.
5 . The method of claim 4 , wherein defining the first region of interest and the second region of interest includes one of: (i) retrieving the first region of interest and the second region of interest from memory, wherein the first region of interest and the second region of interest are manually defined; (ii) defining a polygon in a volume of space, and locating the first region of interest in the first image and the second region of interest in the second image based on the polygon; (iii) defining the first region of interest in the first image and obtaining the second region of interest in the second image using a homography transformation; and (iv) defining at least one of the first region of interest and the second region of interest by selecting a segmentation region.
6 . A system for aligning a first camera of a vehicle with a second camera of the vehicle, comprising: a processor configured to: detect a first set of features in a first image obtained from the first camera; detect a second set of features in a second image obtained from the second camera; determine a first region of interest in the first image; determine a second region of interest in the second image; generate a feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest, wherein minimizing the cost function includes minimizing a descriptor distance for the feature pair, a difference in color of a first pixel in the first image and a second pixel in the second image, the first pixel and the second pixel associated with the feature pair, and a difference in a number of features in a given segmented region of the first region of interest and the second region of interest; align the first camera with the second camera using the feature pair; and control an operation at the vehicle based on the alignment between the first camera and the second camera.
7 . The system of claim 6 , wherein the processor is further configured to determine a transformation matrix between the first camera and the second camera using the feature pairs.
8 . The system of claim 6 , wherein the processor is further configured to store the feature pair in a queue and retrieve the feature pair from the queue using a sliding window.
9 . The system of claim 6 , wherein the processor is further configured to define the first region of interest in the first image and the second region of interest in the second image, the first region of interest corresponding to the second region of interest, detect the first set of features and determining a first set of segmented regions within the first region of interest, and detect the second set of features and determining a second set of segmented regions within the second region of interest.
10 . The system of claim 9 , wherein the processor is further configured to define the first region of interest and the second region of interest by performing one of: (i) retrieving the first region of interest and the second region of interest from memory, wherein the first region of interest and the second region of interest are manually defined; (ii) defining a polygon in a volume of space, and locating the first region of interest in the first image and the second region of interest in the second image based on the polygon; (iii) defining the first region of interest in the first image and obtaining the second region of interest in the second image using a homography transformation; and (iv) defining at least one of the first region of interest and the second region of interest by selecting a segmentation region.
11 . A vehicle, comprising: a first camera; a second camera; and a processor configured to: detect a first set of features in a first image obtained from the first camera; detect a second set of features in a second image obtained from the second camera; determine a first region of interest in the first image; determine a second region of interest in the second image; generating a feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest, wherein minimizing the cost function includes minimizing a descriptor distance for the feature pair, a difference in color of a first pixel in the first image and a second pixel in the second image, the first pixel and the second pixel associated with the feature pair, and a difference in a number of features in a given segmented region of the first region of interest and the second region of interest; align the first camera with the second camera using the feature pair; and control an operation at the vehicle based on the alignment between the first camera and the second camera.
12 . The vehicle of claim 11 , wherein the processor is further configured to determine a transformation matrix between the first camera and the second camera using the feature pairs.
13 . The vehicle of claim 11 , wherein the processor is further configured to store the feature pair in a queue and retrieve the feature pair from the queue using a sliding window.
14 . The vehicle of claim 11 , wherein the processor is further configured to define the first region of interest in the first image and the second region of interest in the second image, the first region of interest corresponding to the second region of interest, detect the first set of features and determine a first set of segmented regions within the first region of interest, and detect the second set of features and determining a second set of segmented regions within the second region of interest.
15 . A method of aligning a first camera of a vehicle with a second camera of the vehicle, comprising: detecting a first set of features in a first image obtained from the first camera; detecting a second set of features in a second image obtained from the second camera; determining a first region of interest in the first image; determining a second region of interest in the second image; generating a feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest, wherein minimizing the cost function includes minimizing a descriptor distance for the feature pair and a difference in a number of features in a given segmented region of the first region of interest and the second region of interest; aligning the first camera with the second camera using the feature pair; and controlling an operation at the vehicle based on the alignment between the first camera and the second camera.
16 . A system for aligning a first camera of a vehicle with a second camera of the vehicle, comprising: a processor configured to: detect a first set of features in a first image obtained from the first camera; detect a second set of features in a second image obtained from the second camera; determine a first region of interest in the first image; determine a second region of interest in the second image; generate a feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest, wherein minimizing the cost function includes minimizing a descriptor distance for the feature pair and a difference in a number of features in a given segmented region of the first region of interest and the second region of interest; align the first camera with the second camera using the feature pair; and control an operation at the vehicle based on the alignment between the first camera and the second camera.
17 . A vehicle, comprising: a first camera; a second camera; and a processor configured to: detect a first set of features in a first image obtained from the first camera; detect a second set of features in a second image obtained from the second camera; determine a first region of interest in the first image; determine a second region of interest in the second image; generating a feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest, wherein minimizing the cost function includes minimizing a descriptor distance for the feature pair and a difference in a number of features in a given segmented region of the first region of interest and the second region of interest; align the first camera with the second camera using the feature pair; and control an operation at the vehicle based on the alignment between the first camera and the second camera.
18 . The method of claim 2 , further comprising decomposing the transformation matrix to obtain a calibration parameter.
19 . The system of claim 7 , wherein the processor is further configured to decompose the transformation matrix to obtain a calibration parameter.
20 . The vehicle of claim 12 , wherein the processor is further configured to decompose the transformation matrix to obtain a calibration parameter.

Description

INTRODUCTION The subject disclosure relates to alignment of cameras in vehicles and, in particular, to a system and method for aligning the coordinate system of cameras using matching regions of interest and matched feature pairs within images from the cameras. Cameras are used on a vehicle to provide information about the surrounding environment that can be used for various operations of the vehicle, such as for planning a motion or trajectory of the vehicle. When multiple cameras are used, images taken from two different cameras can provide conflicting or confusing data if their coordinate systems do not coincide or are not coordinated with each other. Images from fish-eye cameras include optical aberrations that make it difficult to coordinate the cameras using the images. Accordingly, it is desirable to provide a system and method for determining a transformation between cameras having images that include these optical aberrations. SUMMARY In one exemplary embodiment, a method of aligning a first camera of a vehicle with a second camera of the vehicle is disclosed. A first set of features is detected in a first image obtained from the first camera. A second set of features is detected in a second image obtained from the second camera. A first region of interest is determined in the first image. A second region of interest is determined in the second image. A feature pair is generated by matching the first set of features to the second set of features using the first region of interest and the second region of interest. The first camera is aligned with the second camera using the feature pair. An operation at the vehicle is controlled based on the alignment between the first camera and the second camera. In addition to one or more of the features described herein, the method further includes generating the feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest. Minimizing the cost function includes minimizing at least one of a descriptor distance for the feature pair, a difference in color of a first pixel in the first image and a second pixel in the second image, the first pixel and the second pixel associated with the feature pair, and a difference in a number of features in given segmented region of the first region of interest and the second region of interest. The method further includes determining transformation matrix between the first camera and the second camera using the feature pairs. The method further includes storing the feature pair in a queue and retrieving the feature pair from the queue using a sliding window. The method further includes defining the first region of interest in the first image and the second region of interest in the second image, the first region of interest corresponding to the second region of interest, detecting the first set of features and determining a first set of segmented regions within the first region of interest, and detecting the second set of features and determining a second set of segmented regions within the second region of interest. Defining the first region of interest and the second region of interest includes one of retrieving the first region of interest and the second region of interest from memory, wherein the first region of interest and the second region of interest are manually defined, defining a polygon in a volume of space, and locating the first region of interest in the first image and the second region of interest in the second image based on the polygon, defining the first region of interest in the first image and obtaining the second region of interest in the second image using a homography transformation, and defining at least one of the region of interest and the second region of interest by selecting a segmentation region. In another exemplary embodiment, a system for aligning a first camera of a vehicle with a second camera of the vehicle is disclosed. The system includes a processor configured to detect a first set of features in a first image obtained from the first camera, detect a second set of features in a second image obtained from the second camera, determine a first region of interest in the first image, determine a second region of interest in the second image, generate a feature pair by matching the first set of features to the second set of features using the first region of interest and the second region of interest, align the first camera with the second camera using the feature pair, and control an operation at the vehicle based on the alignment between the first camera and the second camera. In addition to one or more of the features described herein, the processor is further configured to generate the feature pair by minimizing a cost function based on the first set of features, the second set of features, the first region of interest and the second region of interest. The processor is further confi