US-12626517-B2 - High-fidelity variation of detected obstacles in autonomous vehicle logs

US12626517B2US 12626517 B2US12626517 B2US 12626517B2US-12626517-B2

Abstract

Method includes inputting log data to perception, including sensor data of objects around AV and information about events occurring with respect to AV and obstacle, information including obstacle parameters, each having actual value detected by sensors; detecting, by perception from sensor data, coordinate region within which obstacle appears; outputting, by perception, data including values for obstacle parameters, each associated with actual or synthetic value; inputting values into planner, to generate values for parameters of AV; during re-run of perception, detecting, by perception, coordinate region in log data and detecting obstacle, as object overlapping coordinate region; outputting, by perception, updated data including updated values for obstacle parameters, each associated with actual value or updated synthetic value; inputting updated values into planner, to generate updated values for parameters of AV; and generating validation result based on values for parameters for AV and updated values for parameters for AV.

Inventors

Allison Liemhetcharat
Chuang Wang

Assignees

DoorDash, Inc.

Dates

Publication Date: 20260512
Application Date: 20241004

Claims (20)

1 . A method performed by one or more processors of a computer system, the method comprising: inputting log data to a perception model, the log data including sensor data of a plurality of objects around an autonomous vehicle (AV) and information about events occurring with respect to the AV and an obstacle of interest among the plurality of objects, wherein the plurality of objects is detected by on-vehicle sensors detecting an environment surrounding the AV, wherein the information comprises obstacle parameters associated with the obstacle of interest, each obstacle parameter having an actual value detected by the on-vehicle sensors; detecting, by the perception model from the environment captured in the sensor data, a coordinate region within which the obstacle of interest appears; outputting, by the perception model, a set of data including a set of values associated with the obstacle parameters, each obstacle parameter being associated with the actual value detected for the obstacle parameter or a synthetic value derived from the actual value; inputting the set of values associated with the obstacle parameters into a planner model, to generate a set of values for parameters associated with a behavior of the AV with respect to the obstacle of interest; during a re-run of the perception model, detecting, by the perception model, the coordinate region in the log data or simulated data derived from the log data; detecting, by the perception model, the obstacle of interest as an object that at least partially overlaps the coordinate region; outputting, by the perception model, an updated set of data that includes an updated set of values associated with the obstacle parameters, each obstacle parameter is associated with the actual value detected for the obstacle parameter or an updated synthetic value derived from the actual value; inputting the updated set of values associated with the obstacle parameters into the planner model, to generate an updated set of values for the parameters associated with the behavior of the AV with respect to the obstacle of interest; generating a validation result based on the set of values for the parameters for the AV and the updated set of values for the parameters for the AV that is obtained during the re-run; and validating the perception model, the planner model, or both based on the validation result.
2 . The method of claim 1 , wherein the detecting the coordinate region further comprises: receiving, through a user interface, a user input selecting the obstacle of interest or an area of the environment captured in the sensor data; and generating, in global coordinates, an oriented bounding box (OBB) surrounding the obstacle of interest or the area, to obtain the coordinate region.
3 . The method of claim 2 , wherein the detecting the obstacle of interest as the object further comprises: during the re-run, detecting, as the obstacle of interest, the object whose centroid is within the OBB.
4 . The method of claim 1 , wherein the detecting the coordinate region further comprises: receiving, through a user interface, a user input selecting the obstacle of interest of the environment captured in the sensor data; computing a centroid of a bounding box of the obstacle of interest in global coordinates; and generating an oriented bounding box (OBB) in the global coordinates to obtain the coordinate region, the OBB surrounding the centroid of the bounding box of the obstacle of interest.
5 . The method of claim 1 , wherein the obstacle parameters include some or all of a start detection time point, a total detection time period, a distance from the AV on an X-axis, a distance from the AV on a Y-axis, and a moving speed.
6 . The method of claim 1 , wherein the synthetic value is varied by a predetermined value from the actual value or randomly chosen from a range of values that are adjacent the actual value.
7 . The method of claim 1 , wherein: the log data further includes information about events occurring with respect to the AV and a plurality of obstacles among the plurality of objects, the obstacle of interest being included in the plurality of obstacles, each obstacle of the plurality of obstacles is associated with a set of parameters, the information in the log data comprises, for each parameter of the set of parameters that is associated with the obstacle, an actual value detected by the on-vehicle sensors, and the perception model outputs a plurality of data sets respectively associated with each obstacle of the plurality of obstacles.
8 . The method of claim 7 , wherein: each of the plurality of data sets for each obstacle includes a set of values for the set of parameters for the obstacle, and each set of values for each set of parameters includes one or more actual values for one or more non-varied parameters, which are parameters that are not varied from the set of parameters of the obstacle, and at least one synthetic value for at least one varied parameter, which is at least one parameter that is varied from the set of parameters of the obstacle.
9 . The method of claim 8 , further comprising: prior to the inputting the set of data for the obstacle parameters into the planner model, filtering the plurality of data sets based on a filtering rule.
10 . The method of claim 9 , wherein the filtering rule specifies to filter out all data except for data associated with the obstacle of interest.
11 . A computer system comprising: a processor; and a non-transitory computer readable medium comprising instructions that, when executed by the processor, cause the processor to perform a method including: inputting log data to a perception model, the log data including (1) sensor data of a plurality of objects around an autonomous vehicle (AV) and (2) information about events occurring with respect to the AV and an obstacle of interest among the plurality of objects, wherein the plurality of objects is detected by on-vehicle sensors detecting an environment surrounding the AV, wherein the information comprises obstacle parameters associated with the obstacle of interest, each obstacle parameter having an actual value detected by the on-vehicle sensors; detecting, by the perception model from the environment captured in the sensor data, a coordinate region within which the obstacle of interest appears; outputting, by the perception model, a set of data including a set of values associated with the obstacle parameters, each obstacle parameter being associated with the actual value detected for the obstacle parameter or a synthetic value derived from the actual value; inputting the set of values associated with the obstacle parameters into a planner model, to generate a set of values for parameters associated with a behavior of the AV with respect to the obstacle of interest; during a re-run of the perception model, detecting, by the perception model, the coordinate region in the log data or simulated data derived from the log data; detecting, by the perception model, the obstacle of interest as an object that at least partially overlaps the coordinate region; outputting, by the perception model, an updated set of data that includes an updated set of values associated with the obstacle parameters, each obstacle parameter is associated with the actual value detected for the obstacle parameter or an updated synthetic value derived from the actual value; inputting the updated set of values associated with the obstacle parameters into the planner model, to generate an updated set of values for the parameters associated with the behavior of the AV with respect to the obstacle of interest; generating a validation result based on the set of values for the parameters for the AV and the updated set of values for the parameters for the AV that is obtained during the re-run; and validating the perception model, the planner model, or both based on the validation result.
12 . The computer system of claim 11 , wherein the detecting the coordinate region further includes: receiving, through a user interface, a user input selecting the obstacle of interest or an area of the environment captured in the sensor data; and generating, in global coordinates, an oriented bounding box (OBB) surrounding the obstacle of interest or the area, to obtain the coordinate region.
13 . The computer system of claim 12 , wherein the detecting the obstacle of interest as the object further includes: during the re-run, detecting, as the obstacle of interest, the object whose centroid is within the OBB.
14 . The computer system of claim 11 , wherein the detecting the coordinate region further includes: receiving, through a user interface, a user input selecting the obstacle of interest of the environment captured in the sensor data; computing a centroid of a bounding box of the obstacle of interest in global coordinates; and generating an oriented bounding box (OBB) in the global coordinates to obtain the coordinate region, the OBB surrounding the centroid of the bounding box of the obstacle of interest.
15 . The computer system of claim 11 , wherein the obstacle parameters include some or all of a start detection time point, a total detection time period, a distance from the AV on an X-axis, a distance from the AV on a Y-axis, and a moving speed.
16 . The computer system of claim 11 , wherein the synthetic value is varied by a predetermined value from the actual value or randomly chosen from a range of values that are adjacent the actual value.
17 . The computer system of claim 11 , wherein: the log data further includes information about events occurring with respect to the AV and a plurality of obstacles among the plurality of objects, the obstacle of interest being included in the plurality of obstacles, each obstacle of the plurality of obstacles is associated with a set of parameters, the information in the log data comprises, for each parameter of the set of parameters that is associated with the obstacle, an actual value detected by the on-vehicle sensors, and the perception model outputs a plurality of data sets respectively associated with each obstacle of the plurality of obstacles.
18 . The computer system of claim 17 , wherein: each of the plurality of data sets for each obstacle includes a set of values for the set of parameters for the obstacle, and each set of values for each set of parameters includes one or more actual values for one or more non-varied parameters, which are parameters that are not varied from the set of parameters of the obstacle, and at least one synthetic value for at least one varied parameter, which is at least one parameter that is varied from the set of parameters of the obstacle.
19 . The computer system of claim 18 , wherein the method further includes: prior to the inputting the set of data for the obstacle parameters into the planner model, filtering the plurality of data sets based on a filtering rule.
20 . The computer system of claim 19 , wherein the filtering rule specifies to filter out all data except for data associated with the obstacle of interest.

Description

BACKGROUND Autonomous vehicles (AVs), also known as self-driving cars, driverless vehicles, and robotic vehicles, are vehicles that use sensors to sense the environment and move without human input. Automation technology enables the AVs to drive on roadways and perceive the surrounding environment accurately and quickly, including but not limited to various obstacles such as pedestrians, animals, signs, trees, traffic lights, etc. In some cases, the AVs can be used to pick up passengers and drive the passengers to selected destinations, pick up a delivery order and drive it to a destination, etc. For example, the AV may drive on a road and sense the events occurring in the environment surrounding the AV, by using sensors. The sensed data, e.g., a log data from the real world, is offloaded from the AV. For example, if a pedestrian is crossing the road in front of the AV, the autonomous system of the AV can control the AV to take an action (e.g., to stop, slow down, etc.) or a human can take control over. Simulations can be performed on the log data to play back the recorded scenario and determine how the AV could control itself if the human did not take over. Based on the simulations, the control commands for the autonomous system of the AV can be developed or improved. SUMMARY Embodiments of the present disclosure provide systems, methods, and apparatuses for addressing the above problems by providing multiple simulation scenarios based on modifications of a real-life log data. Some embodiments include a method performed by one or more processors of a computer system, the method including: inputting log data to a perception model, the log data including sensor data of a plurality of objects around an autonomous vehicle (AV) and information about events occurring with respect to the AV and an obstacle of interest among the plurality of objects, where the plurality of objects is detected by on-vehicle sensors detecting an environment surrounding the AV, where the information includes obstacle parameters associated with the obstacle of interest, each obstacle parameter having an actual value detected by the on-vehicle sensors; detecting, by the perception model from the environment captured in the sensor data, a coordinate region within which the obstacle of interest appears; outputting, by the perception model, a set of data including a set of values associated with the obstacle parameters, each obstacle parameter being associated with the actual value detected for the obstacle parameter or a synthetic value derived from the actual value; inputting the set of values associated with the obstacle parameters into a planner model, to generate a set of values for parameters associated with a behavior of the AV with respect to the obstacle of interest; during a re-run of the perception model, detecting, by the perception model, the coordinate region in the log data or simulated data derived from the log data; detecting, by the perception model, the obstacle of interest as an object that at least partially overlaps the coordinate region; outputting, by the perception model, an updated set of data that includes an updated set of values associated with the obstacle parameters, each obstacle parameter is associated with the actual value detected for the obstacle parameter or an updated synthetic value derived from the actual value; inputting the updated set of values associated with the obstacle parameters into the planner model, to generate an updated set of values for the parameters associated with the behavior of the AV with respect to the obstacle of interest; generating a validation result based on the set of values for the parameters for the AV and the updated set of values for the parameters for the AV that is obtained during the re-run; and validating the perception model, the planner model, or both based on the validation result. Some embodiments include a computer system including: a processor; and a non-transitory computer readable medium including instructions that, when executed by the processor, cause the processor to perform a method including: inputting log data to a perception model, the log data including (1) sensor data of a plurality of objects around an autonomous vehicle (AV) and (2) information about events occurring with respect to the AV and an obstacle of interest among the plurality of objects, where the plurality of objects is detected by on-vehicle sensors detecting an environment surrounding the AV, where the information includes obstacle parameters associated with the obstacle of interest, each obstacle parameter having an actual value detected by the on-vehicle sensors; detecting, by the perception model from the environment captured in the sensor data, a coordinate region within which the obstacle of interest appears; outputting, by the perception model, a set of data including a set of values associated with the obstacle parameters, each obstacle parameter being associated with the act