US-12620303-B2 - Predicting the behavior of road users based on a graph representation of a traffic situation

US12620303B2US 12620303 B2US12620303 B2US 12620303B2US-12620303-B2

Abstract

A method for predicting the behavior of at least one road user in a traffic situation. The method includes: obtaining a graph representation of the traffic situation, wherein nodes represent road users, edges represent interactions between the road users and define an adjacency between road users, each node is associated with a state, and each edge is associated with edge attributes; computing an evolution of the states of the nodes based at least in part on a self-evolution of the state of each considered node that is dependent on this state and mediated by a self-evolution operator; and an interaction of each considered node with other nodes that is dependent on the states of these other nodes and mediated by an interaction operator; and computing a sought property that characterizes the behavior of the at least one road user.

Inventors

Maximilian Zipfl
Achim RETTINGER
Cory Henson
Felix Hertlein
Juergen Luettin
Lavdim Halilaj
Stefan Schmid
Steffen Thoma

Assignees

ROBERT BOSCH GMBH

Dates

Publication Date: 20260505
Application Date: 20230330
Priority Date: 20220831

Claims (15)

1 . A method for predicting a behavior of at least one road user in a traffic situation, comprising the following steps: obtaining a graph representation of the traffic situation, the graph representation having nodes and edges, wherein: the nodes of the graph represent road users, the edges of the graph represent interactions between the road users and define an adjacency between road users, each node of the nodes is associated with a state including a plurality of state variables, and each edge of the edges is associated with a plurality of edge attributes; computing an evolution of the states of the nodes based at least in part on: a self-evolution of the state of each considered node that is dependent on at least the state and mediated by a self-evolution operator; and an interaction of each considered node with other nodes that is dependent on the states of the other nodes and mediated by an interaction operator; and computing at least one sought property that characterizes the behavior of the at least one road user by applying, to the state of the node in the graph representation that corresponds to the at least one road user and/or to a work product derived from the state of the node in the graph representation that corresponds to the at least one road user, an evaluation operator, wherein the self-evolution operator and/or the interaction operator, and/or the evaluation operator includes training a neural network, and wherein the interaction operator includes a neural network with at least two fully connected layers that is applied to the edge attributes of an edge connecting two interacting nodes; wherein computing the at least one sought property includes: updating the self-evolution operator to yield a new state of a node; and predicting, based on the new state, at least one sought property by the evaluation operator of the neural network for each road user in a temporal sequence.
2 . The method of claim 1 , wherein the graph representation is computed from: one or more top-view images of the traffic situation; and/or sensor data recorded with at least one sensor carried by at least one road user; and/or data exchanged by vehicle-to-vehicle communication between road users.
3 . The method of claim 1 , wherein the edge attributes of at least one edge of the edges are configured to encode: a longitudinal relationship between road users that follow each other in a same lane of traffic; or a lateral relationship between road users that travel on adjacent lanes of traffic; or an intersecting relationship between road users travelling on merging and/or intersecting lanes.
4 . The method of claim 1 , wherein the edge attributes of at least one edge of the edges includes: a joint probability that two road users of the road users corresponding to the nodes connected by the edge are on particular lanes; and/or a distance between the two road users corresponding to the nodes connected by the edge, and/or between one or both of the two road users and a point of intersection between the two road users.
5 . The method of claim 1 , wherein the state variables of at least one node of the nodes include: an indication of a type of road user represented by the node; and/or a pollution emission class of the road user represented by the node; and/or dimensions of the road user represented by the node; and/or an estimated mass of the road user represented by the node; and/or a velocity of the road user represented by the node; and/or a state of indicator lights of the road user represented by the node; and/or a presence of cargo and/or passengers on board the road user presented by the node; and/or information about a destination of the road user represented by the node; and/or a context of the road user represented by the node.
6 . The method of claim 1 , wherein the sought property that characterizes the behavior of the at least one road user includes an acceleration of the at least one road user.
7 . The method of claim 1 , wherein: the graph representation is obtained, and evolutions of the states of the nodes are computed, for a temporal sequence of traffic situations; for each traffic situation in the sequence, the computed states of the nodes are inputted into a sequence-processing neural network that is adapted to process sequences of input data; and the evaluation operator is applied to an output of the sequence-processing neural network.
8 . The method of claim 1 , further comprising: determining, based at least in part on the computed property that characterizes the behavior of the at least one road user, an actuation signal; and actuating, using the actuation signal, a vehicle and/or an alarm for alerting a driver of the vehicle.
9 . The method of claim 1 , wherein a relative magnitude of the evolution operator with respect to an aggregated interaction message conveyed by a subset of the nodes indicates an importance the self-evolution is compared with the interaction.
10 . The method of claim 1 , wherein the state variables of the nodes and/or the edge attributes of the edges, include hidden variables without a direct relationship to an observable property of a road user, respectively of a piece of road between road users.
11 . The method of claim 1 , wherein the evaluation operator includes a multi-layer perceptron that is configured as a classifier and/or regressor for the sought property that characterizes the behavior of the at least one road user.
12 . A non-transitory machine-readable data carrier on which is stored a computer program for predicting a behavior of at least one road user in a traffic situation, the computer program, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps: obtaining a graph representation of the traffic situation, the graph representation having nodes and edges, wherein: the nodes of the graph represent road users, the edges of the graph represent interactions between the road users and define an adjacency between road users, each node of the nodes is associated with a state including a plurality of state variables, and each edge of the edges is associated with a plurality of edge attributes; computing an evolution of the states of the nodes based at least in part on: a self-evolution of the state of each considered node that is dependent on at least the state and mediated by a self-evolution operator; and an interaction of each considered node with other nodes that is dependent on the states of the other nodes and mediated by an interaction operator; and computing at least one sought property that characterizes the behavior of the at least one road user by applying, to the state of the node in the graph representation that corresponds to the at least one road user and/or to a work product derived from the state of the node in the graph representation that corresponds to the at least one road user, an evaluation operator, wherein the self-evolution operator and/or the interaction operator, and/or the evaluation operator includes training a neural network, and wherein the interaction operator includes a neural network with at least two fully connected layers that is applied to the edge attributes of an edge connecting two interacting nodes; wherein computing the at least one sought property includes: updating the self-evolution operator to yield a new state of a node; and predicting, based on the new state, at least one sought property by the evaluation operator of the neural network for each road user in a temporal sequence.
13 . The non-transitory machine-readable data carrier of claim 12 , wherein a relative magnitude of the evolution operator with respect to an aggregated interaction message conveyed by a subset of the nodes indicates an importance the self-evolution is compared with the interaction.
14 . One or more computers and/or compute instances configured to predict a behavior of at least one road user in a traffic situation, the one or more computers and/or compute instances configured to: obtain a graph representation of the traffic situation, the graph representation having nodes and edges, wherein: the nodes of the graph represent road users, the edges of the graph represent interactions between the road users and define an adjacency between road users, each node of the nodes is associated with a state including a plurality of state variables, and each edge of the edges is associated with a plurality of edge attributes; compute an evolution of the states of the nodes based at least in part on: a self-evolution of the state of each considered node that is dependent on at least the state and mediated by a self-evolution operator; and an interaction of each considered node with other nodes that is dependent on the states of the other nodes and mediated by an interaction operator; and compute at least one sought property that characterizes the behavior of the at least one road user by applying, to the state of the node in the graph representation that corresponds to the at least one road user and/or to a work product derived from the state of the node in the graph representation that corresponds to the at least one road user, an evaluation operator, wherein the self-evolution operator and/or the interaction operator, and/or the evaluation operator includes training a neural network, and wherein the interaction operator includes a neural network with at least two fully connected layers that is applied to the edge attributes of an edge connecting two interacting nodes; wherein computing the at least one sought property includes: updating the self-evolution operator to yield a new state of a node; and predicting, based on the new state, at least one sought property by the evaluation operator of the neural network for each road user in a temporal sequence.
15 . The one or more computers and/or compute instances of claim 14 , wherein a relative magnitude of the evolution operator with respect to an aggregated interaction message conveyed by a subset of the nodes indicates an importance the self-evolution is compared with the interaction.

Description

CROSS REFERENCE The present application claims the benefit under 35 U.S.C. § 119 of German Patent Application No. DE 10 2022 209 034.9 filed on Aug. 31, 2022, which is expressly incorporated herein by reference in its entirety. FIELD The present invention relates to the determining of the behavior of road users from measurement data that characterize a traffic situation. Knowing the behavior of other road users is, inter alia, important for planning the next actions of an ego-vehicle in traffic. BACKGROUND INFORMATION A vehicle that is being steered through traffic in an at least partially automated manner needs to capture its surroundings and to constantly adapt its own actions to changes in these surroundings. In particular, the vehicle needs to respond to actions of other road users. These actions are usually driven by some interest of the plans of the other road user. However, information about interest or plans for future behavior in traffic are presently exchanged between users only to a very rudimentary extent, namely in the form of turn signals. As a consequence, it is not easy to predict the future behavior of a road user. SUMMARY The present invention provides a method for predicting the behavior of at least one road user in a traffic situation. Herein, the term “road user” comprises all human-piloted and at least partially automated entities that are able to participate in road traffic, such as cars, trucks, buses, trams, motorcycles, bicycles, electric scooters, and humans with or without assisting devices such as roller skates or rollerblades. Also, the term is not limited to users of the driving surface of the road, but also covers users of sidewalks or cycle ways that are also to be considered part of the road. According to an example embodiment of the present invention, in the course of the method, a graph representation of the traffic situation is obtained. Knowledge about the traffic situation that may be processed into a graph representation may be in any form. For example, the graph representation may be computed from one or more top-view images of the traffic situation, which may be recorded by one or more fixed or mobile cameras. For example, some intersections, tunnels, or stretches of road are routinely monitored by cameras to ensure smooth operations, detect safety problems in a tunnel, or dynamically decide whether to open a freeway shoulder as an additional lane to traffic. Alternatively or in combination, sensor data recorded with at least one sensor carried by at least one road user may also be used to compute the graph representation. For example, every car with functionality for at least partially automated driving has at least one camera as a sensor that records camera images. Very frequently, cars are also equipped with radar and/or lidar sensors that produce radar data, respectively lidar data. Combinations of different sensor modalities increase the probability that at least one modality will supply usable information from the environment of the car at any time even if one modality is temporarily not working. For example, a camera may be temporarily unable to supply images because it is directly facing the sun and its sensor is fully saturated. Also, according to an example embodiment of the present invention, data exchanged by vehicle to vehicle, V2V, communication between road users may be used to compute the graph representation. In this manner, the traffic situation may be assessed more accurately from multiple perspectives. Data of all kinds may be aggregated and processed into a graph representation. The more diverse the collected data is, the more complete the information comprised in the graph representation. In the graph representation, the nodes v of the graph represent road users i,j. Each node v is associated with a state h comprising a plurality of state variables. The edges (i,j) of the graph represent interactions between the road users i,j and define an adjacency between road users i,j. Each edge (i,j) is associated with a plurality of edge attributes ei,j. An evolution of the states h of the nodes v is computed. This evolution comprises at least two components, namely a self-evolution of the state h of each considered node v that is dependent on at least this state (and possibly also a history of previous states) and mediated by a self-evolution operator Θ, andan interaction of each considered node v,i with other nodes v,j≠i that is dependent on the states hj≠i of these other nodes v,j≠i and mediated by an interaction operator θedge. At least one sought property ai that characterizes the behavior of at least one road user i is computed by applying an evaluation operator φ to the state his of the node v,i in the graph representation that corresponds to this road user i, and/or a work product derived from such state his. The sought property ai may comprise any suitable quantity and/or quality that characterizes the behavior of the at least one road user i. F