US-12617431-B1 - Risk determination for autonomous vehicle simulations

US12617431B1US 12617431 B1US12617431 B1US 12617431B1US-12617431-B1

Abstract

System, methods, and computer-readable media for determining metrics associated with simulations of proximity events within autonomous vehicle simulations. In some examples, a simulation may be created using driving log data from an autonomous vehicle. Portions of the simulation containing proximity events may be analyzed. Intent times for both the autonomous vehicle and agent vehicles within the simulation may be determined, and the deceleration required for each vehicle to avoid the proximity event may be calculated. These decelerations may be used to determine metrics for the likeliness of the proximity event and the avoidability of the proximity event by the autonomous vehicle. These metrics may be used to prioritize development tasks, assess performance of the simulation, and assess performance of vehicle control systems.

Inventors

Clement Besson
Jonathan Philip Wai Wah Chan
Gary Linscott
Nathan David SHEMONSKI

Assignees

Zoox, Inc.

Dates

Publication Date: 20260505
Application Date: 20220930

Claims (20)

1 . A system comprising: one or more processors; and non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform actions including: accessing a driving log comprising real perception data of a driving scenario within a physical environment; generating a simulation based at least in part on the driving scenario in the driving log, the simulation comprising a simulated autonomous vehicle in a simulated environment, the simulated autonomous vehicle being controlled in the simulation by a control code portion, and at least one simulated agent vehicle controlled at least in part by data from the driving log in the simulated environment; identifying a portion of the simulation wherein the simulated autonomous vehicle has a proximity event with a simulated agent vehicle in a proximity zone; and determining, for the portion of the simulation: a first deceleration that would be required by the simulated agent vehicle to avoid the proximity event; a first value for a first metric representative of the first deceleration; a second deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event; a second value for a second metric representative of a reciprocal of the second deceleration; and a classification of the proximity event as an avoidable or an unavoidable event based on the first value and the second value.
2 . The system of claim 1 , wherein the first deceleration is determined at a time that the simulated autonomous vehicle is on course to occupy the proximity zone; and the second deceleration is determined at a time that the simulated agent vehicle is on course to occupy the proximity zone.
3 . The system of claim 2 , wherein the time that the simulated autonomous vehicle is on course to occupy the proximity zone, and the time that the simulated agent vehicle is on course to occupy the proximity zone are each determined using at least one of a respective: longitudinal intent time; and lateral intent time.
4 . The system of claim 3 , wherein the longitudinal intent time comprises a first time at which an entity's speed indicates intent to occupy the proximity zone, calculated based on a predetermined rate of deceleration; and wherein the lateral intent time comprises a first time at which it is clear the entity's path will lead to the proximity zone.
5 . A method comprising: accessing a driving log comprising real perception data determined for a physical environment; generating a simulation based at least in part on the driving log, the simulation comprising a simulated autonomous vehicle in a simulated environment and a plurality of simulated agent vehicles in the simulated environment; identifying a portion of the simulation wherein the simulated autonomous vehicle has a proximity event with at least one simulated agent vehicle; determining, for the portion of the simulation: a first metric representative of a likeliness of the proximity event occurring; and a second metric representative of a reciprocal of a deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event; and classifying, for the portion of the simulation and based on at least the second metric, the proximity event as an avoidable or an unavoidable event.
6 . The method of claim 5 , wherein: the first metric comprises a measure of a first deceleration required by the simulated agent vehicle to avoid the proximity event at a time that the simulated autonomous vehicle is on course to occupy a proximity zone where the proximity event occurs; and wherein: the second metric comprises a measure of an inverse of a second deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event at a time that the simulated agent vehicle is on course to occupy the proximity zone where the proximity event occurs.
7 . The method of claim 6 , wherein the time that the simulated autonomous vehicle is on course to occupy the proximity zone, and the time that the simulated agent vehicle is on course to occupy the proximity zone are each determined using at least one of a respective: longitudinal intent time; and lateral intent time.
8 . The method of claim 7 , wherein the longitudinal intent time comprises the first time at which an entity's speed indicates intent to occupy the proximity zone, calculated based on a predetermined rate of deceleration; and wherein the lateral intent time comprises the first time at which it is clear the entity's path will lead to the proximity zone.
9 . The method of claim 8 , wherein the predetermined rate of deceleration is one of several predetermined rates of deceleration each corresponding to a respective category of the agent.
10 . The method of claim 9 , wherein the category comprises one of: Car; Large vehicle; Motorcycle; Aircraft; Vehicle with towed load; Bicycle; Animal; and Pedestrian.
11 . The method of claim 6 , wherein the first metric is determined by generating one or more further simulations in which the simulated agent decelerates at different times in order to determine a latest time at which the proximity event is avoided.
12 . The method of claim 6 , wherein the second metric is determined by generating one or more further simulations in which the simulated autonomous vehicle decelerates at different times in order to determine a latest time at which the proximity event is avoided.
13 . The method of claim 5 , further comprising training a machine learning model using at least one data item from a driving log in a case that the proximity event is classified as an avoidable proximity event.
14 . The method of claim 5 , further comprising ranking a plurality of proximity events according to at least one of respective first or second metrics.
15 . The method of claim 13 , further comprising selecting at least one proximity event of the plurality of proximity events for use in evaluating at least one machine learning model, based at least in part on at least one of: the respective first metric; the respective second metric a position of the at least one proximity events within the ranking of the plurality of proximity events; and the classification of the proximity event as avoidable or unavoidable.
16 . The method of claim 5 , further comprising determining a trend across the plurality of simulations according to respective first and second metrics.
17 . One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: accessing a driving log comprising real perception data determined for a physical environment; generating a simulation based at least in part on the driving log, the simulation comprising a simulated autonomous vehicle in a simulated environment and a plurality of simulated agent vehicles in the simulated environment; identifying a portion of the simulation wherein the simulated autonomous vehicle has a proximity event with at least one simulated agent vehicle; determining, for the portion of the simulation: a first metric representative of a likeliness of the proximity event occurring; and a second metric representative of a reciprocal of a deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event; and classifying, for the portion of the simulation and based on at least the second metric, the proximity event as an avoidable or an unavoidable event.
18 . The non-transitory computer-readable media of claim 17 , wherein: the first metric comprises a measure of a first deceleration required by the simulated agent vehicle to avoid the proximity event at a time that the simulated autonomous vehicle is on course to occupy a proximity zone where the proximity event occurs; and wherein: the second metric comprises a measure of an inverse of a second deceleration that would be required by the simulated autonomous vehicle to avoid the proximity event at a time that the simulated agent vehicle is on course to occupy the proximity zone where the proximity event occurs.
19 . The non-transitory computer-readable media of claim 18 , wherein the time that the simulated autonomous vehicle is on course to occupy the proximity zone, and the time that the simulated agent vehicle is on course to occupy the proximity zone are each determined using at least one of a respective: longitudinal intent time; and lateral intent time.
20 . The non-transitory computer-readable media of claim 19 , wherein the longitudinal intent time comprises the first time at which an entity's speed indicates intent to occupy the proximity zone, calculated based on a predetermined rate of deceleration; and wherein the lateral intent time comprises the first time at which it is clear the entity's path will lead to the proximity zone.

Description

BACKGROUND As autonomous vehicles with varying levels of automation have become more widely implemented, manufacturers have been developing, testing and analyzing the systems used in such vehicles. In particular, tests are performed to determine a risk of undesirable interactions between autonomous vehicles and other vehicles in the surrounding environment. As scenarios involving undesirable interactions between vehicles cannot be accurately tested in reality without significant risk of injury and damage, simulated environments have been deployed to simulate these scenarios in the real world. BRIEF DESCRIPTION OF DRAWINGS The detailed description is described with reference to the accompanying figures. The use of the same reference numbers in different figures indicates similar or identical components or features. FIG. 1 is a diagram illustrating a process for determining metrics representative of risk in simulations of autonomous vehicle driving. FIG. 2 is a simplified illustration of an exemplary simulated scenario depicting longitudinal intent times of simulated vehicle models. FIG. 3 is a simplified illustration of an exemplary simulated scenario depicting lateral intent times of simulated vehicle models. FIG. 4A-D are simplified illustrations of exemplary simulated scenarios depicting proximity event scenarios between simulated models. FIG. 5 is a block diagram of an example system for implementing the techniques described herein. DETAILED DESCRIPTION This application relates to methods, systems and computer-readable media that can be used to assess proximity event scenarios determined during simulations of an autonomous vehicle traversing an environment. Due to limitations of the simulations, which will be exemplified later, simulations of the autonomous vehicle that result in a proximity event, where two or more vehicles get closer to each other than a desired threshold distance such as in a proximity event, may or may not represent realistic proximity event scenarios. For example, some proximity events may have been very easily avoidable in a real-world scenario and occur due to limitations in the simulations ability to represent real driver behavior. In various examples described herein, simulations of interactions between an autonomous ‘hero’ vehicle and one or more external ‘agent’ vehicles are assessed using metrics that are representative of aspects of how likely a proximity event between the two entities would have been in real life. A ‘hero’ vehicle may be an autonomous vehicle that is controlled in the simulation by software and/or hardware components. Accordingly, the ‘hero’ vehicle may react differently when changes in software and/or hardware components are applied and a simulation is re-run. An ‘agent’ vehicle may represent a vehicle controlled external to the system described herein, such as by a human driver. The behavior of the ‘agent’ vehicle may have been recorded in a real driving scenario and recorded in log data. The behavior for a hero vehicle may therefore change across multiple simulations based on a particular set of log data representing a particular scenario, as the hero may react differently when different software or hardware parameters are simulated, while behavior relating to an agent vehicle obtained from the log data may not change or may change in a more limited manner, as the behavior of the agent vehicle is determined based on historical log data. Metrics may be computed for the likeliness of the proximity event—this being representative of the deceleration that would be required for an agent vehicle to avoid the proximity event at a given time—and the avoidability of the proximity event, representative of the ability of the hero vehicle to avoid the proximity event at a given time. It is expected that the lower the likeliness metric, the less likely the proximity event is to occur in real-world driving scenarios. Similarly, the avoidability metric may be used to identify scenarios that are more important for an artificial intelligence implementation, such as might control an autonomous vehicle, to address. Over multiple iterations of a simulation, the evolution of a given metric for a given simulated scenario may be examined with different versions of the artificial intelligence implementation to identify whether likelihood of a proximity event is being reduced and to identify more important incidents (those that are more likely to occur in the real world) to study and focus on. Either the likeliness or avoidability metric, or a combination of both, may be used to identify a given proximity event within a scenario as being avoidable or unavoidable. Simulated scenarios may be ranked according to either the likeliness or avoidability metric, and such rankings may provide an indication of the priority of addressing the scenario in question when considering development tasks. In examples, a real-world vehicle, which may be an autonomous vehicle, travelling through a r