US-20260127753-A1 - GENERATING DEPTH INFORMATION IN AUTOMOTIVE IMAGING SYSTEMS

Abstract

A device for image processing includes: one or more memories; and processing circuitry coupled to the one or more memories, wherein the processing circuitry is configured to: receive a first image captured with a first camera and a second image captured with a second camera; determine a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determine a corrected image from the first image based on the rotation component; generate another image from the second image; and generate depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component.

Inventors

• Louis Joseph Kerofsky

Assignees

• QUALCOMM INCORPORATED

Dates

Publication Date

20260507

Application Date

20241107

Claims (20)

1. 1 . A device for image processing, comprising: one or more memories; and processing circuitry coupled to the one or more memories, wherein the processing circuitry is configured to: receive a first image captured with a first camera and a second image captured with a second camera; determine a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determine a corrected image from the first image based on the rotation component; generate another image from the second image; and generate depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component.
2. 2 . The device of claim 1 , wherein the rotation component is a first rotation component and the translation component is a first translation component, wherein the corrected image is a first corrected image, wherein the processing circuitry is configured to determine a second rotation component, and wherein to generate the other image from the second image, the processing circuitry is configured to determine a second corrected image from the second image based on the second rotation component, wherein the second corrected image is the other image generated from the second image.
3. 3 . The device of claim 1 , wherein the rotation component includes a rotational speed indicative of a speed at which the first camera is rotating, and wherein to determine the corrected image from the first image based on the rotation component, the processing circuitry is configured to: determine a time difference between when pixels in the first image and the second image are captured; access optical flow information indicative of an amount of pixel rotational movement in images captured with the first camera for a unit of rotational movement of the first camera; determine a rotational change based on the time difference, the rotational speed, and the optical flow information; and rectify the first image based on the rotational change to determine the corrected image.
4. 4 . The device of claim 1 , wherein to generate the depth information, the processing circuitry is configured to: determine a time difference between when pixels in the first image and the second image are captured; determine a camera baseline between the first camera and the second camera based on the time difference and the translation component; and generate the depth information based on the camera baseline.
5. 5 . The device of claim 4 , wherein to determine the camera baseline between the first camera and the second camera based on the time difference and the translation component, the processing circuitry is configured to: determine a baseline offset based on the time difference and a translation speed of the translation component; and add the baseline offset to an actual distance between the first camera and the second camera to determine the camera baseline.
6. 6 . The device of claim 1 , wherein to determine the rotation component and the translation component, the processing circuitry is configured to determine the rotation component and the translation component based on one or more of: video odometry from the one or more images captured with the first camera; speed and steering wheel angle of a vehicle that includes the first camera and the second camera determined based on the one or more sensors; or sensed motion determined based on the one or more sensors.
7. 7 . The device of claim 1 , wherein to determine the corrected image, the processing circuitry is configured to determine the corrected image from the first image based on the rotation component and without the translation component.
8. 8 . The device of claim 1 , wherein to generate the depth information, the processing circuitry is configured to generate the depth information based on the translation component and without the rotation component.
9. 9 . The device of claim 1 , wherein the processing circuitry is configured to determine disparity information based on the pixels in the corrected image and the corresponding pixels in the other image, and wherein to generate depth information, the processing circuitry is configured to generate the depth information based on the disparity information and the translation component.
10. 10 . The device of claim 1 , wherein the device comprises a vehicle that includes the first camera and the second camera.
11. 11 . The device of claim 1 , wherein the processing circuitry is configured to determine an operating parameter based on the depth information.
12. 12 . The device of claim 11 , wherein the operating parameter includes an operating parameter of a vehicle that includes the first camera and the second camera, and the operating parameter comprises one or more a braking parameter or a path parameter of the vehicle.
13. 13 . A method of image processing, the method comprising: receiving a first image captured with a first camera and a second image captured with a second camera; determining a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determining a corrected image from the first image based on the rotation component; generating another image from the second image; and generating depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component.
14. 14 . The method of claim 13 , wherein the rotation component is a first rotation component and the translation component is a first translation component, wherein the corrected image is a first corrected image, the method further comprising determining a second rotation component, and wherein generating the other image from the second image comprises determining a second corrected image from the second image based on the second rotation component, wherein the second corrected image is the other image generated from the second image.
15. 15 . The method of claim 13 , wherein the rotation component includes a rotational speed indicative of a speed at which the first camera is rotating, and wherein determining the corrected image from the first image based on the rotation component comprises: determining a time difference between when pixels in the first image and the second image are captured; accessing optical flow information indicative of an amount of pixel rotational movement in images captured with the first camera for a unit of rotational movement of the first camera; determining a rotational change based on the time difference, the rotational speed, and the optical flow information; and rectifying the first image based on the rotational change to determine the corrected image.
16. 16 . The method of claim 13 , wherein generating the depth information comprises: determining a time difference between when pixels in the first image and the second image are captured; determining a camera baseline between the first camera and the second camera based on the time difference and the translation component; and generating the depth information based on the camera baseline.
17. 17 . The method of claim 16 , wherein determining the camera baseline between the first camera and the second camera based on the time difference and the translation component comprises: determining a baseline offset based on the time difference and a translation speed of the translation component; and adding the baseline offset to an actual distance between the first camera and the second camera to determine the camera baseline.
18. 18 . The method of claim 13 , wherein determining the rotation component and the translation component comprises determining the rotation component and the translation component based on one or more of: video odometry from the one or more images captured with the first camera; speed and steering wheel angle of a vehicle that includes the first camera and the second camera determined based on the one or more sensors; or sensed motion determined based on the one or more sensors.
19. 19 . The method of claim 13 , further comprising determining an operating parameter based on the depth information.
20. 20 . One or more computer-readable storage media storing instructions thereon that when executed cause one or more processors to: receive a first image captured with a first camera and a second image captured with a second camera; determine a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determine a corrected image from the first image based on the rotation component; generate another image from the second image; and generate depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component.

Description

TECHNICAL FIELD The disclosure relates to image processing including depth data generation. BACKGROUND Example techniques to generate depth information include utilizing two images captured from two different cameras for the same image content. Processing circuitry may determine corresponding pixels in the two images, and disparity between the locations of the corresponding pixels in the two images. Based on the disparity, the processing circuitry may determine depth information (e.g., how far away the objects are from the cameras). SUMMARY In general, this disclosure describes techniques for accounting for rotation and translation movement of cameras of a camera pair for generating depth information, as well as accounting for a time difference between when corresponding pixels are captured due to the use of a rolling shutter. For generating depth information, processing circuitry may be configured to determine corresponding pixels (e.g., pixels for the same physical object captured by the images) of two images that capture the same image content, and disparity between these corresponding pixels. However, the corresponding pixels in the two images may not be captured at the same time due to rolling shutter. Moreover, the position of the corresponding pixels in the two images may be changed if there is movement of the cameras. Together, the timing difference between when corresponding pixels are captured due to rolling shutter, and the movement of the cameras (e.g., rotational or translational movement) may result in generating incorrect depth information. In one or more examples, processing circuitry may be configured to perform per-frame operations that compensate for the rotational changes in the images used for generating depth information due to rolling shutter and camera movement. The processing circuitry may be configured to perform per-pixel operations that compensate for the translational changes in the images used for generating depth information due to rolling shutter and camera movement. In this manner, the generated depth information may more accurately indicate the depth of objects as compared to other techniques. In one example, the disclosure describes a device for image processing, comprising: one or more memories; and processing circuitry coupled to the one or more memories, wherein the processing circuitry is configured to: receive a first image captured with a first camera and a second image captured with a second camera; determine a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determine a corrected image from the first image based on the rotation component; generate another image from the second image; and generate depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component. In one example, the disclosure describes a method of image processing, the method comprising: receiving a first image captured with a first camera and a second image captured with a second camera; determining a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determining a corrected image from the first image based on the rotation component; generating another image from the second image; and generating depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component. In one example, the disclosure describes one or more computer-readable storage media storing instructions thereon that when executed cause one or more processors to: receive a first image captured with a first camera and a second image captured with a second camera; determine a rotation component and a translation component of the first camera based on one or more of images captured with the first camera or one or more sensors; determine a corrected image from the first image based on the rotation component; generate another image from the second image; and generate depth information based on pixels in the corrected image and corresponding pixels in the other image and the translation component. The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a diagram of an example vehicle in accordance with the techniques of this disclosure for object detection in accordance with one or more examples described in this disclosure. FIGS. 2A and 2B illustrate examples of images captured with cameras. FIG. 3 is a block diagram of a device configured to perform one or more of the example techniques described in this disclosure. FIG. 4 is a flowchart illustrating a method of image processing according to one or more example techniques d