US-12617342-B1 - Multi camera image display

Abstract

A vehicle includes at least one exterior facing camera and a viewing screen in communication with a controller. The camera includes a red green blue (RGB) imaging sensor and a near infrared (NIR) imaging sensor. An ambient light sensor is disposed on the vehicle and detects a magnitude of ambient lighting in an exterior environment. The controller stores instructions for: Receiving a RGB image from the RGB imaging sensor at a time t, receiving a NIR image from the NIR imaging sensor at the time t, and receiving a lumens value of the ambient light at the time t. Preprocessing the RGB image into a processed RGB image using one of a plurality of preprocessing techniques dependent upon the lumens magnitude of the ambient lighting. Fusing the processed RGB image and the NIR image into a single viewing image using a neural network. Displaying the viewing image.

Inventors

• Jonglee Park
• Julien P. Mourou
• Charles R. Quinn
• Manoj Kumar Sharma

Assignees

• GM Global Technology Operations LLC

Dates

Publication Date

20260505

Application Date

20241206

Claims (20)

1. 1 . A vehicle comprising: at least one exterior facing camera in communication with a controller, wherein the exterior facing camera includes a red green blue (RGB) imaging sensor and a near infrared (NIR) imaging sensor; a viewing screen in communication with the controller; an ambient light sensor disposed on the vehicle and configured to detect a magnitude of ambient lighting in an exterior environment, the ambient light sensor being in communication with the controller; the controller including a processor and a memory, the memory storing instructions for causing the processor to perform operations comprising: receiving a RGB image from the RGB imaging sensor at a time t, receiving a NIR image from the NIR imaging sensor at the time t, and receiving a lumens value of ambient light at the time t; preprocessing the RGB image into a processed RGB image using one of a plurality of preprocessing techniques, wherein the preprocessing technique used is dependent upon the magnitude of the lumens value of the ambient lighting; fusing the processed RGB image and the NIR image into a single viewing image using a neural network; and displaying a viewing image on the viewing screen.
2. 2 . The vehicle of claim 1 , wherein the neural network is a convolutional neural network trained via a training data set including a first set of NIR images and RGB images captured at a low lighting condition, a second set of NIR images and RGB images captured at an intermediate light condition, and a third set of NIR images and RGB images captured at a high light condition.
3. 3 . The vehicle of claim 2 , wherein the first set of NIR images and RGB images is captured at an ambient lighting condition below a first threshold, and wherein RGB images in the first set of NIR images and RGB images are processed using an enhancement function.
4. 4 . The vehicle of claim 2 , wherein the second set of NIR images and RGB images is captured at an ambient lighting condition above a first threshold and below a second threshold, and wherein RGB images in the second set of NIR images and RGB images are not processed.
5. 5 . The vehicle of claim 2 , wherein the third set of NIR images and RGB images is captured at an ambient lighting condition above a first threshold and above a second threshold, and wherein RGB images in the third set of NIR images and RGB images are processed using a tone mapping function.
6. 6 . The vehicle of claim 2 , wherein the training data set includes color features extracted from the RGB images in the first, second and third training data set and contrast features extracted from the NIR images in the first, second and third training data set.
7. 7 . The vehicle of claim 6 , wherein the color features are extracted via a first loss function according to: L perceptual ( Y 2 , Y ) = 1 C × H × W ⁢  ϕ ⁡ ( Y 2 ) - ϕ ⁡ ( Y )  2 2 where L is the extracted features, Y 2 is a final fused image output, and Y is a ground truth image from the RGB imaging sensor, C is a set of color channels defining each image, H is a height of each image and W is a width of each image; and wherein the contrast features are extracted via a second loss function according to: L n ⁢ i ⁢ r ( Y 2 , N ) = 1 C × H × W ⁢  ϕ ⁡ ( Y 2 ) - ϕ ⁡ ( N )  2 2 where N is the original NIR image from the NIR imaging sensor.
8. 8 . The vehicle of claim 7 , wherein the first loss function and the second loss function are combined into a third loss function according to: L=L perceptual ( Y 2 ,Y )+λ L nir ( Y 2 ,N ) where, 0<λ<1λ is a weighting parameter between 0 and 1 and wherein λ is dependent on a lumens magnitude of the ambient light detected by the ambient light sensor.
9. 9 . The vehicle of claim 8 , wherein A has an elevated value at high and low lumens magnitudes.
10. 10 . The vehicle of claim 1 , wherein the controller further includes a mutual camera functionality test.
11. 11 . The vehicle of claim 1 , wherein the NIR imaging sensor and the RGB imaging sensor are disposed proximate each other.
12. 12 . The vehicle of claim 1 , wherein the viewing screen is proximate a side view mirror, and wherein the viewing image is a side view image.
13. 13 . The vehicle of claim 1 , wherein the NIR imaging sensor and the RGB imaging sensor define a field of view at a side of the vehicle, and wherein the viewing image is a side view mirror replacement image.
14. 14 . A method for providing a viewing image to a vehicle operator comprising: receiving a RGB image from a RGB imaging sensor within a camera at a time t, receiving a NIR image from a NIR imaging sensor within the camera at the time t, and receiving a lumens value of the ambient light at the time t; preprocessing the RGB image into a processed RGB image using one of a plurality of preprocessing techniques, wherein the preprocessing technique used is dependent upon the lumens magnitude of the ambient lighting; fusing the processed RGB image and the NIR image into a single viewing image using a neural network; and displaying the viewing image on the viewing screen.
15. 15 . The method of claim 14 , wherein the neural network is a convolutional neural network trained via a training data set including a first set of NIR images and RGB images captured at a low lighting condition, a second set of NIR images and RGB images captured at an optimum light condition, and a third set of NIR images and RGB images captured at a high light condition.
16. 16 . The method of claim 15 , wherein the first set of NIR images and RGB images is captured at an ambient lighting condition below a first threshold, and wherein RGB images in the first set of NIR images and RGB images are processed using an enhancement function.
17. 17 . The method of claim 15 , wherein the second set of NIR images and RGB images is captured at an ambient lighting condition above a first threshold and below a second threshold, and wherein RGB images in the second set of NIR images and RGB images are not processed.
18. 18 . The method of claim 15 , wherein the training data set includes color features extracted from the RGB images in the first, second and third training data set and contrast features extracted from the NIR images in the first, second and third training data set.
19. 19 . The method of claim 14 , wherein the viewing image is a side view mirror replacement image.
20. 20 . The method of claim 14 , wherein the viewing image is a side view mirror supplement and the viewing screen is disposed proximate a side view mirror supplemented by the viewing image.

Description

The subject disclosure relates to vehicles, and in particular to a system for generating a single image using multiple cameras in a vehicle. Vehicles include viewing mirrors, such as rear facing side view mirrors, to provide vehicle operator's with a view that would otherwise be impossible or impractical to achieve from the position of the vehicle operator. Some vehicles include cameras, such as those used in a vehicle vision system, that are adjacent to traditionally placed mirrors or in place of the traditionally placed mirrors. In such cases, the driver is provided with a view generated by the camera(s). The cameras, however, are limited by lighting conditions. When the lighting is too bright, the image produced can be oversaturated. Similarly, when the lighting is too dark, the image produced can be undersaturated. When images are oversaturated or undersaturated, it can be difficult to distinguish between distinct elements present in the image. This, in turn, limits the usefulness of the images as mirror replacements or as mirror supplements. It is desirable to provide a system that compensates for oversaturation and/or undersaturation and provides the user with a legible image generated by the camera(s) without requiring the user to manually adjust a saturation of the camera images. SUMMARY In one exemplary embodiment a vehicle includes at least one exterior facing camera in communication with a controller. The camera includes a red green blue (RGB) imaging sensor and a near infrared (NIR) imaging sensor. A viewing screen is in communication with the controller. An ambient light sensor is disposed on the vehicle and is configured to detect a magnitude of ambient lighting in an exterior environment. The ambient light sensor is in communication with the controller. The controller includes a processor and a memory. The memory stores instructions for causing the processor to perform the operations of: Receiving a RGB image from the RGB imaging sensor at a time t, receiving a NIR image from the NIR imaging sensor at the time t, and receiving a lumens value of the ambient light at the time t. Preprocessing the RGB image into a processed RGB image using one of a plurality of preprocessing techniques, where the preprocessing technique used is dependent upon the lumens magnitude of the ambient lighting. Fusing the processed RGB image and the NIR image into a single viewing image using a neural network. Displaying the viewing image on the viewing screen. In addition to one or more of the features described herein the neural network is a convolutional neural network trained via a training data set including a first set of NIR images and RGB images captured at a low lighting condition, a second set of NIR images and RGB images captured at an optimum light condition, and a third set of NIR images and RGB images captured at a high light condition. In addition to one or more of the features described herein the first set of NIR images and RGB images is captured at an ambient lighting condition below a first threshold, and wherein RGB images in the first set of NIR images and RGB images are processed using an enhancement function. In addition to one or more of the features described herein the second set of NIR images and RGB images is captured at an ambient lighting condition above a first threshold and below a second threshold, and wherein RGB images in the second set of NIR images and RGB images are not processed. In addition to one or more of the features described herein the third set of NIR images and RGB images is captured at an ambient lighting condition above a first threshold and above a second threshold, and wherein RGB images in the third set of NIR images and RGB images are processed using a tone mapping function. In addition to one or more of the features described herein the training data set includes color features extracted from the RGB images in the first, second and third training data set and contrast features extracted from the NIR images in the first, second and third training data set. In addition to one or more of the features described herein the color features are extracted via a first loss function according to: Lperceptual(Y2,Y)=1C×H×W⁢ϕ⁡(Y2)-ϕ⁡(Y)22where L is the extracted features, Y2 is a final fused image output, and Y is a ground truth image from the RGB imaging sensor, C is a set of color channels defining each image, H is a height of each image and W is a width of each image; and wherein the contrast features are extracted via a second loss function according to: Ln⁢i⁢r(Y2,N)=1C×H×W⁢ϕ⁡(Y2)-ϕ⁡(N)22where N is the original NIR image from the NIR imaging sensor. In addition to one or more of the features described herein the first loss function and the second loss function are combined into a third loss function according to: L=Lperceptual(Y2, Y)+λ Lnir(Y2, N) where, 0<λ<1, λ is a weighting parameter between 0 and 1 and wherein λ is dependent on a lumens magnitude of the ambient ligh