Search

JP-7856721-B2 - Method and system for evaluating autonomous driving systems

JP7856721B2JP 7856721 B2JP7856721 B2JP 7856721B2JP-7856721-B2

Inventors

  • 北島 慎之助
  • パッターウット・マリーヌアン
  • 権藤 正樹

Assignees

  • イーソル株式会社

Dates

Publication Date
20260511
Application Date
20240926

Claims (10)

  1. In an autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has learned human driving behavior through machine learning, Multiple sets of actual driving data from human drivers and model driving data from driving based on the output results of the aforementioned probability model are used. The first step is for the computer to calculate the difference between two data points selected from multiple actual driving data, The second step involves the computer calculating the difference between one data point selected from multiple real-world driving data points and one data point selected from multiple model driving data points. An evaluation step in which a computer compares the difference obtained in the first step with the difference obtained in the second step to evaluate the similarity of the trajectories, Having, Evaluation methods for autonomous driving systems.
  2. The aforementioned probability model was constructed using machine learning with the aforementioned multiple real-world driving data. A method for evaluating an autonomous driving system according to claim 1.
  3. In the first and second steps described above, the RMSE (Root Mean Square Error) of the two selected data is calculated. A method for evaluating an autonomous driving system according to claim 1.
  4. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the average value of the differences obtained by performing the first step multiple times with the average value of the differences obtained by performing the second step multiple times. A method for evaluating an autonomous driving system according to claim 1.
  5. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the standard deviation of the difference obtained by performing the first step multiple times with the standard deviation of the difference obtained by performing the second step multiple times. A method for evaluating an autonomous driving system according to claim 1.
  6. In an autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has learned human driving behavior through machine learning, Multiple sets of actual driving data from human drivers and model driving data from driving based on the output results of the aforementioned probability model are used. By running the program, The first step is to calculate the difference between two data points selected from multiple actual driving data, The second step involves calculating the difference between one data point selected from multiple real-world driving data points and one data point selected from multiple model driving data points. An evaluation step to evaluate the similarity of the trajectories by comparing the difference obtained in the first step with the difference obtained in the second step, Execute An evaluation system for autonomous driving systems.
  7. The aforementioned probability model was constructed using machine learning with the aforementioned multiple real-world driving data. An evaluation system for an autonomous driving system according to claim 6.
  8. In the first and second steps described above, the RMSE (Root Mean Square Error) of the two selected data is calculated. An evaluation system for an autonomous driving system according to claim 6.
  9. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the average value of the differences obtained by performing the first step multiple times with the average value of the differences obtained by performing the second step multiple times. An evaluation system for an autonomous driving system according to claim 6.
  10. In the evaluation step, the degree of similarity of the trajectories is evaluated by comparing the standard deviation of the difference obtained by performing the first step multiple times with the standard deviation of the difference obtained by performing the second step multiple times. An evaluation system for an autonomous driving system according to claim 6.

Description

This disclosure relates to a method and system for evaluating an autonomous driving system for a vehicle that outputs non-deterministic signals or parameters using a probabilistic model that has been trained on human driving behavior, and more particularly to a method and system for evaluating the performance of the probabilistic model. In recent years, vehicles such as automobiles have been equipped with systems that automatically or semi-automatically control the vehicle using computers. For example, there are automobiles equipped with Advanced Driver-Assistance Systems (ADAS), which use various sensors to gather information about the vehicle's surroundings and control the vehicle based on this information to assist driving. In typical advanced autonomous driving systems, rules for controlling the vehicle are pre-defined, and control actions are determined by comparing these rules with the vehicle's state (the model that generates such control actions is referred to here as a "rule-based model"). However, human driving is known to be probabilistic, and it is difficult to mimic human driving using pre-defined rules. Therefore, the vehicle control actions in rule-based models may differ significantly from human driving. Such differing control may lead to a loss of passenger comfort or cause passengers to feel unsafe due to control actions that exceed human predictions. To address these problems, research is being conducted on techniques that collect data from actual vehicle driving and generate models using machine learning (these models are referred to here as "probabilistic models"). Probabilistic models aim to mimic human driving behavior and output parameters used for vehicle control non-deterministically (i.e., the same input will not produce the same output). Using such probabilistic models, it may be possible to achieve driving behavior close to that of human drivers. By the way, when vendors develop probabilistic models like the ones described above, it's necessary to evaluate the model by determining how well it can mimic human driving. In this regard, Patent Document 1 discloses a method for evaluating the driving data of an autonomous vehicle. Specifically, Patent Document 1 proposes a computer-based method for evaluating the performance of a dynamic simulator used to simulate the behavior of an autonomous vehicle (ADV), the method comprising: receiving multiple states of the ADV at multiple points in time and multiple control commands given to the ADV at those multiple points in time; generating multiple predicted positions of the simulated trajectory of the ADV based on the multiple states and the multiple control commands; receiving multiple actual positions of the ground truth trajectory of the ADV generated by giving the control commands to the ADV at those multiple points in time; and generating evaluation metrics to measure the similarity between the multiple predicted positions of the simulated trajectory and the multiple actual positions of the ground truth trajectory at those multiple points in time. Japanese Patent Publication No. 2022-58566 This is a block diagram showing the outline configuration of a vehicle equipped with an autonomous driving system.This is a block diagram showing the outline configuration of the evaluation system for autonomous driving systems.This diagram illustrates the steps performed by the evaluation system.This is a flowchart illustrating an example of the process performed by the evaluation system.(a) A table showing actual driving data numerically, and (b) An image diagram plotting the actual driving data.(a) A diagram illustrating an example of the first step of calculating the difference between two data points selected from multiple actual driving data; (b) A conceptual diagram of the difference obtained by performing the first step multiple times; (c) A conceptual diagram of the mean and standard deviation of the difference.(a) A diagram illustrating an example of the second step of calculating the difference between one data point selected from multiple actual driving data points and one data point selected from multiple model driving data points; (b) A conceptual diagram of the difference obtained by performing the second step multiple times; (c) A conceptual diagram of the mean and standard deviation of the difference.This table explains the numerical values obtained using the evaluation method. Embodiments of the present invention will be described with reference to the figures. The evaluation system 200 according to this embodiment is for evaluating an autonomous driving system for a vehicle using a probabilistic model. The probabilistic model is constructed by machine learning human driving behavior and outputs non-deterministic signals or parameters (the same input does not produce the same output). The autonomous driving system is configured to be able to drive autonomously using the output of this probabilistic model. As for the vehicle, for e