EP-4736137-A1 - SCENE RELIABILITY

EP4736137A1EP 4736137 A1EP4736137 A1EP 4736137A1EP-4736137-A1

Abstract

A computer implemented method is provided. The method, comprises: receiving from an autonomous vehicle sensor data, wherein the sensor data is indicative of an environment in which the vehicle is currently located or was previously located. The method further comprises receiving or determining additional data associated with at least one of: the sensor data, the vehicle, or the environment, wherein the additional data is different from the sensor data. The method further comprises displaying, on a display, an output comprising a representation of the sensor data. The method further comprises determining, based at least in part on the additional data, a reliability metric, the reliability metric being indicative of how reliable the sensor data is at representing the environment in which the vehicle is located at a current time. The method further comprises causing the output on the display to be based at least in part on the reliability metric.

Inventors

CHAO, LIANG
CHOUDRY, MUHAMMAD UMAR
CID FERNANDEZ, Gerardo
GOGNA, RAVI
Goldman, Meredith James
ORECCHIO, Paul

Assignees

Zoox, Inc.

Dates

Publication Date: 20260506
Application Date: 20240613

Claims (20)

1. A system comprising: one or more processors; and one or more non-transitory computer readable media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving, at the system, from an autonomous vehicle: video data comprising video frame data, wherein the video frame data is indicative of an environment in which the vehicle is currently located or was previously located; causing a display to output a representation of the video frame data; receiving or determining, at the system, additional data associated with at least one of the vehicle or the environment; determining a time difference associated with the video frame data; determining, based at least in part on the additional data and the time difference associated with the video frame data, a reliability metric, the reliability metric being indicative of how reliable the video frame data is at representing an environment in which the vehicle is located at a current time; and causing the output on the display to be based at least in part on the reliability metric.
2. The system of claim 1 , wherein causing the output on the display to be based at least in part on the reliability metric comprises causing the display to output a graphical item, wherein the presence of and/or characteristics of the graphical item are based on the reliability metric.
3. The system of claim 2, wherein the characteristics include at least one of: a form, size, color, position or opacity of the graphical item.
4. The system of claim 1, wherein the time difference is based on one of: a time difference between a time at which the video frame data was recorded by the vehicle and the current time; a time difference between a time at which the video frame data was sent by the vehicle and the current time; a time difference between a time at which the video frame data was received by the system and the current time; or a time difference between a time at which the representation of the video frame data was output by the display and the current time; wherein a smaller time difference results in, or contributes to, a higher reliability metric and a greater time difference results in, or contributes to, a lower reliability metric; and wherein a higher reliability metric indicates that the video frame data is more reliable at representing the environment in which the vehicle is located at the current time than a lower reliability metric.
5. The system of claim 1, wherein data associated with the vehicle comprises at least one of: a speed of the vehicle, wherein a lower speed results in, or contributes to, a higher reliability metric and a higher speed results in, or contributes to, a lower reliability metric, wherein a higher reliability metric indicates that the video frame data is more reliable at representing the environment in which the vehicle is located at the current time than a lower reliability metric; a distance travelled by the vehicle, wherein a lower distance results in, or contributes to, a higher reliability metric and a higher distance results in, or contributes to, a lower reliability metric; or a condition of one or more sensors of the vehicle, wherein a better condition results in, or contributes to, a higher reliability metric and a poorer condition results in, or contributes to, a lower reliability metric.
6. The system of claim 1, wherein data associated with the environment comprises at least one of: a number of non-stationary objects in the environment, wherein a lower number of non- stationary objects results in, or contributes to, a higher reliability metric and a higher number of non-stationary objects results in, or contributes to, a lower reliability metric, wherein a higher reliability metric indicates that the video frame data is more reliable at representing the environment in which the vehicle is located at the current time than a lower reliability metric; a distance between the vehicle and one or more non-stationary objects in the environment, wherein a greater distance between the vehicle and the one or more non-stationary objects in the environment results in, or contributes to, a higher reliability metric and a smaller distance between the vehicle and the one or more non-stationary objects in the environment results in, or contributes to, a lower reliability metric; or a characteristic and/or state of one or more objects in the environment, wherein the characteristic and/or state of each object results in, or contributes to, a higher or lower reliability metric based on the object.
7. A computer implemented method, comprising: receiving from an autonomous vehicle: sensor data, wherein the sensor data is indicative of an environment in which the vehicle is currently located or was previously located; receiving or determining additional data associated with at least one of: the sensor data, the vehicle, or the environment; displaying, on a display, an output comprising a representation of the sensor data; determining, based at least in part on the additional data, a reliability metric, the reliability metric being indicative of how reliable the sensor data is at representing the environment in which the vehicle is located at a current time; and causing the output on the display to be based at least in part on the reliability metric.
8. The method of claim 7, wherein determining the reliability metric comprises determining the reliability metric from a reliability metric scale comprising a plurality of reliability metrics including at least a minimum reliability metric and a maximum reliability metric, the maximum reliability metric indicating that the sensor data is more reliable at representing the environment in which the vehicle is located at the current time than the minimum reliability metric.
9. The method of claim 8, wherein causing the output on the display to be based at least in part on the reliability metric comprises displaying, on the display, a graphical item, wherein the graphical item is based on the reliability metric, wherein a graphical item associated with the minimum reliability metric has a different form, size, color, position and/or opacity of the compared to a graphical item associated with the maximum reliability metric.
10. The method of claim 7, wherein the additional data comprises at least one of: data associated with the vehicle, wherein the data associated with the vehicle comprises at least one of: a speed of the vehicle or a distance travelled by the vehicle or a location of the vehicle or a condition of one or more sensors of the vehicle; or data associated with the environment, wherein data associated with the environment comprises at least one of: data associated with a number of non-stationary objects in the environment or a distance between the vehicle and one or more non-stationary objects in the environment, or a characteristic and/or state of one or more objects in the environment.
11. The method of claim 7, wherein: data associated with the sensor data comprises a timestamp, wherein the timestamp is based on one of: a time at which the sensor data was recorded by the vehicle, or a time at which the sensor data was sent by the vehicle, or a time at which the sensor data was received, or a time at which the representation of the sensor data was output by the display; the method further comprises determining, based on the timestamp and the current time, a time difference associated with the sensor data; and wherein determining, based on the additional data, the reliability metric, comprises determining the reliability metric based on the time difference.
12. The method of claim 7, further comprising: sending, to the vehicle, command data, wherein the command data comprises an instruction for the vehicle, wherein the vehicle implements the instruction based on one of: the reliability metric or a reliability of a network connection.
13. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a system, cause the system to perform operations comprising: receiving, at the system, from an autonomous vehicle: sensor data, wherein the sensor data is indicative of an environment in which the vehicle is currently located or was previously located; causing a display to output a representation of the sensor data; receiving or determining, at the system, additional data associated with at least one of: the sensor data, the vehicle, or the environment; determining, based at least in part on the additional data, a reliability metric, the reliability metric being indicative of how reliable the sensor data is at representing the environment in which the vehicle is located at a current time; and causing the output on the display to be based at least in part on the reliability metric.
14. The one or more non-transitory computer-readable media of claim 13, wherein causing the output on the display to be based at least in part on the reliability metric, comprises at least one of: causing the display to output a graphical item, wherein the graphical item is based on the reliability metric; causing the display to output a representation of the additional data; causing the display to cease the output of the representation of the sensor data; or causing the display to dim the output of the representation of the sensor data.
15. The one or more non-transitory computer-readable media of claim 14, wherein causing the display to output a graphical item, comprises one of: causing display to output the graphical item to obscure at least part of the representation of the sensor data; or causing display to output the graphical item adjacent the representation of the sensor data.
16. The one or more non-transitory computer-readable media of claim 14, wherein the operations further comprise: comparing the reliability metric to a threshold reliability metric and wherein the graphical item is based on the comparison.
17. The one or more non-transitory computer-readable media of claim 13, wherein the additional data indicates that the vehicle is stationary, and the operations further comprise causing the display to output a 3D representation of the environment comprising one or more non- stationary objects; wherein causing the output on the display to be based at least in part on the reliability metric comprises causing the display to display one or more graphical indications to indicate that at least one of: the 3D representation of the environment should be viewed to determine positions of the one or more non-stationary objects or the representation of the sensor data should be viewed to determine positions of one or more stationary objects.
18. The one or more non-transitory computer-readable media of claim 13, wherein causing the output on the display to be based at least in part on the reliability metric comprises causing the display to indicate within the representation of the sensor data, at least one of: (i) positions of one or more non-stationary objects which may no longer be in the indicated positions, or (ii) predicted trajectories or predicted positions of the one or more non-stationary objects, or (iii) a region within the environment based on the detection of an object having a particular classification and/or state.
19. The one or more non-transitory computer-readable media of claim 13, wherein the operations further comprise determining positions of one or more non-stationary objects based on the additional data and wherein causing the output on the display to be based at least in part on the reliability metric comprises causing the display to indicate within the representation of the sensor data, positions of the one or more non-stationary objects.
20. The one or more non-transitory computer-readable media of claim 13, wherein the additional data associated with the environment comprises historical sensor data associated with the environment, and wherein causing the output on the display to be based at least in part on the reliability metric comprises: causing the display to cease the output of the representation of the sensor data and causing the display to output a representation of the historical sensor data, wherein the historical sensor data is indicative of the environment in which the vehicle is located at the current time.

Description

SCENE RELIABILITY BACKGROUND [0001] Autonomous and partially autonomous vehicles are increasingly being tested and used not only for convenience, but also to improve road safety. Autonomous vehicles may have a combination of different sensors that can be used to detect nearby objects to help the vehicle navigate through the environment. BRIEF DESCRIPTION OF DRAWINGS [0002] The detailed description is described with reference to the accompanying figures. The use of the same reference numbers in different figures indicates similar or identical components or features. [0003] FIG. 1 is a pictorial diagram of system, including a vehicle and remote system for monitoring the vehicle, according to an example; [0004] FIG. 2 is a pictorial diagram of a first example output on a display at the remote system; [0005] FIG. 3 is a pictorial diagram of a second example output on a display at the remote system; [0006] FIGs. 4A-12 are pictorial diagrams of further examples of outputs on a display at the remote system; [0007] FIG. 13 depicts a flow chart of a method according to an example; and [0008] FIG. 14 is a block diagram of an example vehicle system. DETAILED DESCRIPTION [0009] This application relates to systems, methods and computer-readable media for determining how reliable sensor data, such as video data, is for representing the current location of an autonomous vehicle, where the sensor data has been recorded by the vehicle at an earlier time. Once the reliability of the sensor/video data has been determined, a visual output of the sensor data can be adapted depending upon the reliability. For example, a vehicle may send video data to a remote system so that it can be displayed to a human operator for monitoring purposes or in case the operator needs to provide instructions to help navigate the vehicle through the environment. In addition to this display, adapting the display based on the reliability of the sensor/video data can inform the operator whether the sensor/video data can be used as a reliable reference or if should be used with caution or disregarded, thereby improving awareness of the operator and resulting in improved safety. [0010] In locations with good connectivity (such as good cellular connectivity), video or other sensor data may be streamed from the vehicle, to the remote system, in real or near-real time. Under such connectivity conditions, where the video data is sent from the vehicle and displayed at the remote system in a short time, the video data may be considered reliable at representing the current environment around the vehicle. However, when connectivity is poor, the transmission of the video data may be restricted, meaning that subsequent frames of the video data may not be sent by the vehicle and/or may not be received at the remote system. In this situation the remote system may not receive updated video data for some time. A previous video frame may therefore be displayed to a human operator, despite this being considered “old” or outdated. As such, the last frame of video frame data received by the system may no longer accurately represent the current environment around the vehicle. For example, the vehicle may have moved from the location represented in the video data, or one or more objects (such as other vehicles or pedestrians) may have moved relative to the vehicle. As a result, it is desirable to indicate to the operator that the video data may be less reliable or unreliable. [0011] The present disclosure therefore relates to determining a reliability metric associated with the sensor/video data (such as a value, numerical value, or some other indicator), where the reliability metric is indicative of how reliable sensor/video data is at representing the current environment around the vehicle. For example, if the latest video frame received by the remote system was recorded only a short time ago, such as less than 2 seconds ago, it may be classified as more reliable (such as having a higher reliability metric) than if the video frame data was recorded 10 seconds ago. As another example, if the vehicle is stationary or has moved only a short distance since video frame data stopped being sent to the remote system, the video frame data may be classified as more reliable than if the vehicle had moved a large distance. As a further example, if the vehicle is in a remote area with few cars and pedestrians, the video frame data may be determined as more reliable than if the vehicle was located in a busy city because the environment around the vehicle is less likely to change, or change to a lesser degree when there are fewer moving objects. [0012] In some examples, the reliability of the video frame data can be determined based on two or more parameters, such as two or more of the following parameters: (i) a time difference between when the video frame data was recorded and when the video frame data was received by the remote system, (ii) a distance travelled by the vehicle, (