US-12619489-B2 - Detecting anomalies in embedded artificial intelligence systems

Abstract

In some examples, a system receives, from a mobile apparatus, at least mobile apparatus location data and artificial intelligence (AI) output data. The system determines a location corresponding to the mobile apparatus location data, and accesses, based at least on the location, past AI output data stored in a data structure and associated with the location. The system compares the received AI output data with the past AI output data stored in the data structure for the location. Based at least on comparing the received AI output data with the past AI output data associated with the location, the system sends an instruction to the mobile apparatus to cause the mobile apparatus to perform at least one action.

Inventors

• Hiroaki ITSUJI
• Subrata Kumar KUNDU

Assignees

• HITACHI, LTD.

Dates

Publication Date

20260505

Application Date

20240201

Claims (20)

1. 1 . A system comprising: one or more processors configured by executable instructions to perform operations comprising: receiving, from a vehicle, at least vehicle location data and artificial intelligence (AI) output data generated by an AI algorithm on the vehicle; determining a road segment corresponding to the vehicle location data, the AI output data received from the vehicle including object recognition data indicating an object recognized by the AI algorithm on the vehicle at the road segment; accessing, based at least on the road segment, past AI output data stored in a database and associated with the road segment, the past AI output data stored in the database including object recognition data indicating objects recognized at a plurality of road segments, including the road segment, by AI algorithms of a plurality of vehicles that previously traversed the plurality of road segments; comparing the received AI output data including the object recognized by the AI algorithm on the vehicle with the past AI output data including the objects recognized and stored in the database for the plurality of vehicles that previously traversed the road segment; based at least on comparing the received AI output data with the past AI output data associated with the road segment, determining an indication of an AI algorithm anomaly for the AI algorithm of the vehicle; and based at least on determining the indication of the AI algorithm anomaly, sending an instruction to the vehicle to cause the vehicle to perform at least one action including at least one of: changing an operating speed of the vehicle; or navigating the vehicle to a maintenance location.
2. 2 . The system as recited in claim 1 , the operations further comprising: receiving, from the vehicle, a vehicle identifier; and based at least on the received vehicle identifier, determining a sensor configuration for the vehicle, wherein the comparing the received AI output data with the past AI output data for the road segment further comprises taking into consideration the sensor configuration for the vehicle during the comparing.
3. 3 . The system as recited in claim 1 , wherein comparing the received AI output data with the past AI output data comprises detecting at least one difference between movable objects identified in the received AI output data and movable objects identified in the past AI output data.
4. 4 . The system as recited in claim 3 , wherein, based at least on detecting the at least one difference, the instruction sent to the vehicle includes an instruction for the vehicle to perform periodic diagnosis of the AI algorithm corresponding to the AI output data to check for the AI algorithm anomaly.
5. 5 . The system as recited in claim 1 , wherein comparing the received AI output data with the past AI output data comprises detecting at least one difference between stationary objects identified in the received AI output data and stationary objects identified in the past AI output data.
6. 6 . The system as recited in claim 5 , wherein, based at least on detecting the at least one difference, the instruction sent to the vehicle includes an instruction for the vehicle to perform at least one of a cause identification or recovery process for the AI algorithm corresponding to the AI output data to check for the AI algorithm anomaly.
7. 7 . The system as recited in claim 1 , wherein: comparing subsequently received AI output data with the past AI output data comprises failing to identify another indicator of an AI algorithm anomaly; and based at least on failing to identify another indicator of an AI algorithm anomaly, another instruction sent to the vehicle includes an instruction for the vehicle to periodically send additional AI output data to continue to monitor for an AI algorithm anomaly.
8. 8 . The system as recited in claim 7 , the operations further comprising: comparing a confidence score associated with the subsequently received AI output data with a confidence score threshold; and based at least on the confidence score associated with the subsequently received AI output data satisfying the confidence score threshold, storing the subsequently received AI output data in the database in association with the road segment.
9. 9 . A method comprising: receiving, by one or more processors, from a mobile apparatus, at least mobile apparatus location data and artificial intelligence (AI) output data generated by an AI algorithm on the mobile apparatus; determining a location corresponding to the mobile apparatus location data, the AI output data received from the mobile apparatus including object recognition data indicating an object recognized by the AI algorithm on the mobile apparatus at the location; accessing, based at least on the location, past AI output data stored in a database and associated with the location, the past AI output data stored in the database including object recognition data indicating objects recognized at a plurality of locations, including the location, by AI algorithms of a plurality of mobile apparatuses that previously traversed the plurality of locations; comparing the received AI output data including the object recognized by the AI algorithm on the mobile apparatus with the past AI output data including the objects recognized and stored in the database for the plurality of mobile apparatuses that previously traversed the location; based at least on comparing the received AI output data with the past AI output data associated with the location, determining an indication of an AI algorithm anomaly for the AI algorithm of the mobile apparatus; and based at least on determining the indication of the AI algorithm anomaly, sending an instruction to the mobile apparatus to cause the mobile apparatus to perform at least one action including at least one of: changing an operating speed of the mobile apparatus; or navigating the mobile apparatus to a maintenance location.
10. 10 . The method as recited in claim 9 , further comprising: receiving, from the mobile apparatus, a mobile apparatus identifier; and based at least on the received mobile apparatus identifier, determining a sensor configuration for the mobile apparatus, wherein the comparing the received AI output data with the past AI output data for the location further comprises taking into consideration the sensor configuration for the mobile apparatus during the comparing.
11. 11 . The method as recited in claim 9 , wherein comparing the received AI output data with the past AI output data comprises detecting at least one difference between movable objects identified in the received AI output data and movable objects identified in the past AI output data.
12. 12 . The method as recited in claim 11 , wherein, based at least on detecting the at least one difference, the instruction sent to the mobile apparatus includes an instruction for the mobile apparatus to perform periodic diagnosis of the AI algorithm corresponding to the AI output data to check for the AI algorithm anomaly.
13. 13 . The method as recited in claim 9 , wherein comparing the received AI output data with the past AI output data comprises detecting at least one difference between stationary objects identified in the received AI output data and stationary objects identified in the past AI output data.
14. 14 . The method as recited in claim 13 , wherein, based at least on detecting the at least one difference, the instruction sent to the mobile apparatus includes an instruction for the mobile apparatus to perform at least one of a cause identification or recovery process for the AI algorithm corresponding to the AI output data to check for the AI algorithm anomaly.
15. 15 . The method as recited in claim 9 , wherein: comparing subsequently received AI output data with the past AI output data comprises failing to identify an indicator of another AI algorithm anomaly, and based at least on failing to identify another indicator of an AI algorithm anomaly, another instruction sent to the mobile apparatus includes an instruction for the mobile apparatus to periodically send additional AI output data to continue to monitor for an AI algorithm anomaly.
16. 16 . The method as recited in claim 15 , further comprising: comparing a confidence score associated with the subsequently received AI output data with a confidence score threshold; and based at least on the confidence score associated with the subsequently received AI output data satisfying the confidence score threshold, storing the subsequently received AI output data in the database in association with the location.
17. 17 . One or more non-transitory computer-readable media storing instructions executable by one or more processors to cause the one or more processors to perform operations comprising: receiving, from a mobile apparatus, at least mobile apparatus location data and artificial intelligence (AI) output data generated by an AI algorithm on the mobile apparatus; determining a location corresponding to the mobile apparatus location data, the AI output data received from the mobile apparatus including object recognition data indicating an object recognized by the AI algorithm on the mobile apparatus at the location; accessing, based at least on the location, past AI output data stored in a database and associated with the location, the past AI output data stored in the database including object recognition data indicating objects recognized at a plurality of locations, including the location, by AI algorithms of a plurality of mobile apparatuses that previously traversed the plurality of locations; comparing the received AI output data including the object recognized by the AI algorithm on the mobile apparatus with the past AI output data including the objects recognized and stored in the database for the plurality of mobile apparatuses that previously traversed the location; and based at least on comparing the received AI output data with the past AI output data associated with the location, determining an indication of an AI algorithm anomaly for the AI algorithm of the mobile apparatus; and based at least on determining the indication of the AI algorithm anomaly, sending an instruction to the mobile apparatus to cause the mobile apparatus to perform at least one action including at least one of: changing an operating speed of the mobile apparatus; or navigating the mobile apparatus to a maintenance location.
18. 18 . The one or more non-transitory computer-readable media as recited in claim 17 , the operations further comprising: receiving, from the mobile apparatus, a mobile apparatus identifier; and based at least on the received mobile apparatus identifier, determining a sensor configuration for the mobile apparatus, wherein the comparing the received AI output data with the past AI output data for the location further comprises taking into consideration the sensor configuration for the mobile apparatus during the comparing.
19. 19 . The one or more non-transitory computer-readable media as recited in claim 17 , wherein: comparing the received AI output data with the past AI output data comprises detecting at least one difference between movable objects identified in the received AI output data and movable objects identified in the past AI output data; and based at least on detecting the at least one difference, the instruction sent to the mobile apparatus includes an instruction for the mobile apparatus to perform periodic diagnosis of the AI algorithm corresponding to the AI output data to check for the AI algorithm anomaly.
20. 20 . The one or more non-transitory computer-readable media as recited in claim 17 , wherein: comparing the received AI output data with the past AI output data comprises detecting at least one difference between stationary objects identified in the received AI output data and stationary objects identified in the past AI output data; and based at least on detecting the at least one difference, the instruction sent to the mobile apparatus includes an instruction for the mobile apparatus to perform at least one of a cause identification or recovery process for the AI algorithm corresponding to the AI output data to check for the AI algorithm anomaly.

Description

BACKGROUND Semi-autonomous vehicle systems or otherwise advanced driver assistance systems (ADAS), and self-driving systems or otherwise autonomous driving (AD) systems are systems that automate or otherwise enhance vehicle control for improved safety, automated navigation, and the like. Such systems may employ embedded onboard computing devices referred to as “electronic control units” (ECUs), which may include AD/ADAS ECUs, powertrain ECUs, chassis ECUs, and so forth. For example, an AD/ADAS ECU may be configured to execute domain specific functionalities, e.g., perception, localization, navigation, vehicle control, and the like. To perform these functions, artificial intelligence (AI) algorithms (e.g., machine learning models) may be implemented on AD/ADAS ECUs or other vehicle ECUs. AI algorithms may include deep neural networks (DNNs) and/or various other types of artificial neural networks and/or various other types of machine learning models. For example, AI algorithms may be used in the ECUs of AD/ADAS vehicles for realizing a highly accurate and safe automated driving experience. For instance, an AI application may be used in various roles, such as in object detection and mission-critical controls, e.g., for performing emergency braking, lane changes, evasive maneuvering, and the like. A malfunction in an AI algorithm upon which a vehicle depends may lead to severe accidents. Accordingly, while the use of AI algorithms for enabling AD/ADAS in vehicles is expected to be beneficial, ensuring that these systems are operating correctly and as intended can improve vehicle safety. SUMMARY In some implementations, a system receives, from a mobile apparatus, at least mobile apparatus location data and artificial intelligence (AI) output data. The system determines a location corresponding to the mobile apparatus location data, and accesses, based at least on the location, past AI output data stored in a data structure and associated with the location. The system compares the received AI output data with the past AI output data stored in the data structure for the location. Based at least on comparing the received AI output data with the past AI output data associated with the location, the system sends an instruction to the mobile apparatus to cause the mobile apparatus to perform at least one action. BRIEF DESCRIPTION OF THE DRAWINGS The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features. FIG. 1 illustrates a logical configuration of an example system for performing efficient monitoring of AI algorithms according to some implementations. FIG. 2 illustrates an example hardware configuration of the system of FIG. 1 according to some implementations. FIG. 3 illustrates an example of diagnosis levels according to some implementations. FIG. 4 is a flow diagram illustrating an example process for performing diagnostics on an AI algorithm according to some implementations. FIG. 5 illustrates an example of identifying an AI algorithm likely to have an AI anomaly from among a plurality of AI algorithms according to some implementations. FIG. 6 illustrates an example of reducing the number of false positives for detecting AI anomalies according to some implementations. FIG. 7 illustrates an example of determining whether to add the AI application output data to the historical AI application output database according to some implementations. DESCRIPTION Some implementations herein are directed to techniques and arrangements for detecting anomalies that may occur in AI algorithms, such as may be used by AI applications that are executed for ADAS and/or AD systems employed in vehicles and/or in computing devices of other types of mobile apparatuses that may employ AI algorithms, such as aerial drones or other types of aerial vehicles, marine drones and other types of marine vehicles, submarine drones and other types of submarine vehicles, mobile autonomous robots and semiautonomous robots, and so forth. For instance, AI applications may include applications that execute an AI algorithm for generating an output or result. The examples herein are able to detect indicators of AI algorithm anomalies for various AI-embedded systems. The examples herein include detecting indicators of AI algorithm anomalies on the cloud side based on current and/or past AI application outputs, and further based on the vehicle's onboard sensor configuration information. Examples of AI application outputs may include detected object identifiers, detected object locations, corresponding confidence scores, and the like. For example, when a current AI application output is inconsistent with the historical AI application output at the same location, and the confidence level of the current