US-12617414-B2 - Aligning sensor data for vehicle applications

Abstract

This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. A method is disclosed for aligning top-down features from two sensor arrangements and generating vehicle control instructions. The method includes receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement. The method further includes determining a first set of top-down and a second set of top-down features based on the sensor data. A transformation is determined based on the first set of top-down features and the second set of top-down features to align the second set of top-down features with the first set of top-down features. Finally, vehicle control instructions for a vehicle are determined based on the transformation. Other aspects and features are also claimed and described.

Inventors

• Kiran BANGALORE RAVI
• Varun Ravi Kumar
• Senthil Kumar Yogamani

Assignees

• QUALCOMM INCORPORATED

Dates

Publication Date

20260505

Application Date

20230810

Claims (20)

1. 1 . A method comprising: receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement, wherein the first sensor arrangement is associated with a first vehicle type and the second sensor arrangement is associated with a second vehicle type different from the first vehicle type, and wherein at least one of the first sensor data and the second sensor data include both image data and position data; determining, based on a first set of sensor parameters of the first sensor arrangement and a second set of sensor parameters of the second sensor arrangement, first differences between the first sensor arrangement and the second sensor arrangement; determining, based on the first sensor data, a first set of top-down features and, based on the second sensor data, a second set of top-down features; determining, based on the first differences, the first set of top-down features, and the second set of top-down features, a transformation to align the second set of top-down features with the first set of top-down features within a top view representation; and determining, based on the transformation, vehicle control instructions for a vehicle of the second vehicle type.
2. 2 . The method of claim 1 , further comprising: determining, based on the first set of top-down features and the second set of top-down features, second differences between the first set of top-down features and the second set of top-down features.
3. 3 . The method of claim 2 , wherein the second differences are determined as one or more stochastic distances between the first sensor arrangement and the second sensor arrangement based on different sensor values, different feature values, and different feature locations.
4. 4 . The method of claim 2 , further comprising: determining the transformation based on the second differences, a first set of latent features for the first sensor data, and a second set of latent features for the second sensor data.
5. 5 . The method of claim 4 , wherein determining the first set of top-down features and determining the second set of top-down features comprises: determining the first set of latent features based on the first sensor data; determining the first set of top-down features based on the first set of latent features; determining the second set of latent features based on the second sensor data; and determining the second set of top-down features based on the second set of latent features.
6. 6 . The method of claim 4 , wherein the transformation is determined by training a variational autoencoder to minimize statistical differences between the first set of top-down features and the second set of top-down features, wherein the statistical differences are determined based on the second differences, the first differences, the first set of latent features, and the second set of latent features.
7. 7 . The method of claim 2 , wherein the transformation is determined in response to determining that (i) the second differences exceed a first predetermined threshold, (ii) the first differences exceed a second predetermined threshold, or (iii) a combination thereof.
8. 8 . The method of claim 1 , wherein the first differences are determined as a weighted difference between the first sensor arrangement and the second sensor arrangement based on a different number of sensors, different field of view coverage for sensors, different sensor ranges, or a combination thereof.
9. 9 . A system comprising: a processing system including one or more processors and one or more memories storing instructions which, when executed by the processing system, cause the processing system to perform operations comprising: receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement, wherein the first sensor arrangement is associated with a first vehicle type and the second sensor arrangement is associated with a second vehicle type different from the first vehicle type, and wherein at least one of the first sensor data and the second sensor data include both image data and position data; determining, based on a first set of sensor parameters of the first sensor arrangement and a second set of sensor parameters of the second sensor arrangement, first differences between the first sensor arrangement and the second sensor arrangement; determining, based on the first sensor data, a first set of top-down features and, based on the second sensor data, a second set of top-down features; determining, based on the first differences, the first set of top-down features, and the second set of top-down features, a transformation to align the second set of top-down features with the first set of top-down features within a top view representation; and determining, based on the transformation, vehicle control instructions for a vehicle of the second vehicle type.
10. 10 . The system of claim 9 , wherein the operations further comprise: determining, based on the first set of top-down features and the second set of top-down features, second differences between the first set of top-down features and the second set of top-down features.
11. 11 . The system of claim 10 , wherein the second differences are determined as one or more stochastic distances between the first sensor arrangement and the second sensor arrangement based on different sensor values, different feature values, and different feature locations.
12. 12 . The system of claim 10 , wherein the operations further comprise: determining the transformation based on the second differences, a first set of latent features for the first sensor data, and a second set of latent features for the second sensor data.
13. 13 . The system of claim 12 , wherein determining the first set of top-down features and determining the second set of top-down features comprises: determining the first set of latent features based on the first sensor data; determining the first set of top-down features based on the first set of latent features; determining the second set of latent features based on the second sensor data; and determining the second set of top-down features based on the second set of latent features.
14. 14 . The system of claim 12 , wherein the transformation is determined by training a variational autoencoder to minimize statistical differences between the first set of top-down features and the second set of top-down features, wherein the statistical differences are determined based on the second differences, the first differences, the first set of latent features, and the second set of latent features.
15. 15 . The system of claim 10 , wherein the transformation is determined in response to determining that (i) the second differences exceed a first predetermined threshold, (ii) the first differences exceed a second predetermined threshold, or (iii) a combination thereof.
16. 16 . The system of claim 9 , wherein the first differences are determined as a weighted difference between the first sensor arrangement and the second sensor arrangement based on a different number of sensors, different field of view coverage for sensors, different sensor ranges, or a combination thereof.
17. 17 . A non-transitory, computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform operations comprising: receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement, wherein the first sensor arrangement is associated with a first vehicle type and the second sensor arrangement is associated with a second vehicle type different from the first vehicle type, and wherein at least one of the first sensor data and the second sensor data include both image data and position data; determining, based on a first set of sensor parameters of the first sensor arrangement and a second set of sensor parameters of the second sensor arrangement, first differences between the first sensor arrangement and the second sensor arrangement; determining, based on the first sensor data, a first set of top-down features and, based on the second sensor data, a second set of top-down features; determining, based on the first differences, the first set of top-down features, and the second set of top-down features, a transformation to align the second set of top-down features with the first set of top-down features within a top view representation; and determining, based on the transformation, vehicle control instructions for a vehicle of the second vehicle type.
18. 18 . The non-transitory, computer-readable medium of claim 17 , wherein the operations further comprise: determining, based on the first set of top-down features and the second set of top-down features, second differences between the first set of top-down features and the second set of top-down features.
19. 19 . The non-transitory, computer-readable medium of claim 18 , wherein the second differences are determined as one or more stochastic distances between the first sensor arrangement and the second sensor arrangement based on different sensor values, different feature values, and different feature locations.
20. 20 . The non-transitory, computer-readable medium of claim 18 , wherein the operations further comprise: determining the transformation based on the second differences, a first set of latent features for the first sensor data, and a second set of latent features for the second sensor data.

Description

TECHNICAL FIELD Aspects of the present disclosure relate generally to driver-operated or driver-assisted vehicles, and more particularly, to methods and systems suitable for supplying driving assistance or for autonomous driving. INTRODUCTION Vehicles take many shapes and sizes, are propelled by a variety of propulsion techniques, and carry cargo including humans, animals, or objects. These machines have enabled the movement of cargo across long distances, movement of cargo at high speed, and movement of cargo that is larger than could be moved by human exertion. Vehicles originally were driven by humans to control speed and direction of the cargo to arrive at a destination. Human operation of vehicles has led to many unfortunate incidents resulting from the collision of vehicle with vehicle, vehicle with object, vehicle with human, or vehicle with animal. As research into vehicle automation has progressed, a variety of driving assistance systems have been produced and introduced. These include navigation directions by GPS, adaptive cruise control, lane change assistance, collision avoidance systems, night vision, parking assistance, and blind spot detection. BRIEF SUMMARY OF SOME EXAMPLES The following summarizes some aspects of the present disclosure to provide a basic understanding of the discussed technology. This summary is not an extensive overview of all contemplated features of the disclosure and is intended neither to identify key or critical elements of all aspects of the disclosure nor to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in summary form as a prelude to the more detailed description that is presented later. Human operators of vehicles can be distracted, which is one factor in many vehicle crashes. Driver distractions can include changing the radio, observing an event outside the vehicle, and using an electronic device, etc. Sometimes circumstances create situations that even attentive drivers are unable to identify in time to prevent vehicular collisions. Aspects of this disclosure, provide improved systems for assisting drivers in vehicles with enhanced situational awareness when driving on a road. In one aspect, a method is provided that includes receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement, wherein at least one of the first sensor data and the second sensor data include both image data and position data. The method further includes determining, based on the first sensor data, a first set of top-down features and, based on the second sensor data, a second set of top-down features and determining, based on the first set of top-down features and the second set of top-down features, a transformation to align the second set of top-down features with the first set of top-down features. The method also includes determining, based on the transformation, vehicle control instructions for a vehicle. In another aspect, a system is provided that includes a processing system including one or more processors and one or more memories storing instructions which, when executed by the processing system, cause the processing system to perform operations. The operations include receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement, wherein at least one of the first sensor data and the second sensor data include both image data and position data. The operations further include determining, based on the first sensor data, a first set of top-down features and, based on the second sensor data, a second set of top-down features and determining, based on the first set of top-down features and the second set of top-down features, a transformation to align the second set of top-down features with the first set of top-down features. The operations also include determining, based on the transformation, vehicle control instructions for a vehicle. In an additional aspect, a non-transitory, computer-readable medium is provided storing instructions which, when executed by a processor, cause the processor to perform operations. The operations include receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement, wherein at least one of the first sensor data and the second sensor data include both image data and position data. The operations further include determining, based on the first sensor data, a first set of top-down features and, based on the second sensor data, a second set of top-down features and determining, based on the first set of top-down features and the second set of top-down features, a transformation to align the second set of top-down features with the first set of top-down features. The operations also include determining, based on the transformation, vehicle control instructions for a vehicle. In a further aspec