US-12617436-B2 - Trajectory prediction from multi-sensor fusion

Abstract

Methods and systems for predicting a trajectory an autonomous vehicle (AV) are disclosed. A method includes generating, based on sensor data from a sensing system of the AV, one or more embeddings, generating, using a machine learning model (MLM) and the one or more embeddings, one or more predicted future trajectories for the AV, and causing, using the one or more predicted future trajectories, a planning system of the AV to generate an update to a current trajectory of the AV.

Inventors

• Yu Ouyang
• Daniel Ho
• Alper Ayvaci
• Tencia Lee
• Bayram Safa CICEK

Assignees

• WAYMO LLC

Dates

Publication Date

20260505

Application Date

20231122

Claims (20)

1. 1 . A method, comprising: generating, based on sensor data from a sensing system of an autonomous vehicle (AV), one or more embeddings characterizing an environment around the AV; generating, using a machine learning model (MLM) and the one or more embeddings as input to the MLM, one or more outputs of the MLM, the one or more outputs comprising a predicted future trajectory of the AV, and determined driving environment conditions of the AV, wherein the predicted future trajectory of the AV comprises a plurality of coordinates indicating predicted future locations of the AV at respective future times; and causing, using the predicted future trajectory, a planning system of the AV to generate an update to a current trajectory of the AV.
2. 2 . The method of claim 1 , wherein: the sensor data comprises data from a plurality of sensor devices of the sensing system of the AV; and generating, based on the sensor data from the sensing system of the AV, the one or more embeddings comprises combining the data from the plurality of sensor devices into an embedding from the one or more embeddings.
3. 3 . The method of claim 2 , wherein the plurality of sensor devices of the sensing system comprises at least one of: a camera; a radar unit; or a lidar unit.
4. 4 . The method of claim 1 , wherein the one or more embeddings comprise: a first embedding that characterizes the environment around the AV at a first time; and a second embedding that characterizes the environment around the AV at a second time.
5. 5 . The method of claim 4 , wherein: the first time comprises a current time; and the second time comprises a time before the first time.
6. 6 . The method of claim 1 , further comprising generating, using the MLM and the one or more embeddings, a predicted future trajectory of an object in the environment around the AV.
7. 7 . The method of claim 1 , wherein: each of the plurality of coordinates includes a time indicating a future time at which the AV is predicted to be located at a respective coordinate; and a predetermined amount of time separates consecutive coordinates of the plurality of coordinates.
8. 8 . The method of claim 7 : further comprising connecting the plurality of coordinates using one or more lines to form a polyline; and wherein causing the planning system to generate the update to the current trajectory of the AV comprises providing the polyline to the planning system.
9. 9 . A system, comprising: a memory; and one or more processing devices, coupled to the memory, configured to perform operations comprising: generating one or more embeddings based on sensor data from a sensing system of an autonomous vehicle (AV), wherein each embedding of the one or more embeddings characterizes an environment around the AV; generating, using a machine learning model (MLM) and the one or more embeddings as input to the MLM, one or more outputs of the MLM, the one or more outputs comprising a plurality of predicted future trajectories of the AV, and determined driving environment conditions of the AV, wherein a predicted future trajectory of the plurality of predicted future trajectories of the AV comprises a plurality of coordinates indicating predicted future locations of the AV at respective future times; and causing, using the plurality of predicted future trajectories, a planning system of the AV to generate an update to a current trajectory of the AV.
10. 10 . The system of claim 9 , wherein the one or more embeddings comprises at least ten embeddings.
11. 11 . The system of claim 9 , wherein: the one or more embeddings comprise a predetermined number of embeddings; and in response to adding a new embedding to the one or more embeddings, removing an oldest embedding from the one or more embeddings.
12. 12 . The system of claim 9 , wherein: an embedding from the one or more embeddings comprises an array of numerical values; and a first portion of the array of numerical values comprises a numerical value characterizing a drivable surface.
13. 13 . The system of claim 12 , wherein a second portion of the array of numerical values comprises a numerical value characterizing an orientation of a head of a pedestrian.
14. 14 . The system of claim 12 , wherein a third portion of the array of numerical values comprises a numerical value characterizing an orientation of a head of a driver of a vehicle.
15. 15 . The system of claim 12 , wherein a fourth portion of the array of numerical values comprises a numerical value characterizing a state of a traffic light.
16. 16 . The system of claim 9 , wherein the operations further comprise generating, using the MLM and the one or more embeddings: a plurality of predicted future trajectories of an object in the environment around the AV; and an orientation of the object at one or more locations along the predicted future trajectories of the object.
17. 17 . A system, comprising: a memory; and one or more processing devices, coupled to the memory, configured to perform operations comprising: generating, based on sensor data from a sensing system of an autonomous vehicle (AV), one or more embeddings characterizing an environment around the AV; generating, using a machine learning model (MLM) and the one or more embeddings as input to the MLM, one or more outputs of the MLM, the one or more outputs comprising a first probability distribution, and determined driving environment conditions of the AV, wherein the first probability distribution comprises: a plurality of predicted future trajectories of the AV, wherein a predicted future trajectory of the plurality of predicted future trajectories of the AV comprises a plurality of coordinates indicating predicted future locations of the AV at respective future times, and for each predicted future trajectory of the plurality of predicted future trajectories of the AV, a probability; and causing, using the first probability distribution, a planning system of the AV to generate an update to a current trajectory of the AV.
18. 18 . The system of claim 17 , wherein the plurality of predicted future trajectories of the AV comprises at least three predicted future trajectories.
19. 19 . The system of claim 17 , wherein the operations further comprise generating, by the MLM and based on the one or more embeddings, a second probability distribution for a predicted future trajectory of an object in the environment around the AV.
20. 20 . The system of claim 17 , wherein: the operations further comprise training the MLM using a dataset that comprises a plurality of records associated with historical data previously recorded for a second AV; and each record of the dataset comprises: sensor data from a sensing system of the second AV in an environment around the second AV, a location history of the second AV in the environment around the second AV, a location history of an object in the environment around the second AV, and a ground truth comprising a trajectory of the second AV in the environment around the second AV.

Description

TECHNICAL FIELD The instant specification generally relates to autonomous vehicles. More specifically, the instant specification relates to trajectory prediction from multi-sensor fusion. BACKGROUND Autonomous vehicles (AVs), whether fully autonomous or partially self-driving, often operate by sensing an outside environment with various sensors (e.g., radar, optical, audio, humidity, etc.). This outside environment may include other objects in the environment, some of which are mobile. Such objects can include other vehicles, cyclists, pedestrians, animals, etc. AVs should avoid colliding with these mobile objects, and avoiding such collisions often involves predicting a path an object may take. AVs can use machine learning (ML) models to predict these paths, which allows the AVs to select a safe and efficient driving path and trajectory for the AV by avoiding the predicted paths of the other objects. BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure is illustrated by way of examples, and not by way of limitation, and can be more fully understood with references to the following detailed description when considered in connection with the figures, in which: FIG. 1 depicts a block diagram of an example autonomous vehicle (AV) capable of utilizing trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. FIG. 2A depicts a top-down view illustrating an example driving environment, in accordance with some implementations of the present disclosure. FIG. 2B depicts an example embedding used by systems and methods for trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. FIG. 3 depicts a flowchart diagram of an example method for trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. FIG. 4A depicts an example dataflow for trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. FIG. 4B depicts an example dataflow for trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. FIG. 5A depicts a top-down view illustrating an example driving environment with a coordinate set, in accordance with some implementations of the present disclosure. FIG. 5B depicts a top-down view illustrating an example driving environment with a polyline, in accordance with some implementations of the present disclosure. FIG. 6 depicts a flowchart diagram of an example method for trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. FIG. 7 depicts a block diagram of an example computer device capable of trajectory prediction from multi-sensor fusion, in accordance with some implementations of the present disclosure. SUMMARY In one implementation, disclosed is a method for trajectory prediction from multi-sensor fusion. The method includes generating, based on sensor data from a sensing system of an autonomous vehicle (AV), one or more embeddings characterizing an environment around the AV. The method includes generating, by a machine learning model (MLM) and using the one or more embeddings, a predicted future trajectory of the AV. The method includes causing, using the predicted future trajectory, a planning system of the AV to generate an update to a current trajectory of the AV. In another implementation, disclosed is a system for trajectory prediction from multi-sensor fusion. The system includes a memory and one or more processing devices coupled to the memory and configured to perform operations. The operations include generating multiple of embeddings based on sensor data from a sensing system of an AV. Each embedding characterizes an environment around the AV. The operations include generating, by an MLM and based on the multiple embeddings, multiple predicted future trajectories of the AV. The operations include providing the multiple predicted future trajectories to a planning system of the AV to generate an update to a current trajectory of the AV. In another implementation, disclosed is a system for trajectory prediction from multi-sensor fusion. The system includes a memory and one or more processing devices coupled to the memory and configured to perform operations. The operations include generating, based on sensor data from a sensing system of an AV, one or more embeddings characterizing an environment around the AV. The operations include generating, by an MLM and using the one or more embeddings, a probability distribution. The probability distribution includes multiple predicted future trajectories of the AV. The probability distribution includes, for each predicted future trajectory, a probability. The operations include causing, using the probability distribution, a planning system of the AV to generate an update to a current trajectory of the AV. DETAILED DESCRIPTIO