US-12619416-B2 - Continuous integration and continuous delivery of artificial intelligence machine learning components using metamorphic relations

Abstract

A system for CI/CD of AI/ML based components includes a cloud infrastructure which receives multiple new requirements from a vehicle designer or a developer, with the new requirements adapted for artificial intelligence/machine learning (AI/ML) based components of a vehicle. A dataset is provided. A metamorphic relations (MR) module receives input information from the dataset and sends MR information to the dataset. A components requirements database includes the new requirements in addition to existing requirements for the AI/ML based components. The MR module also receives components requirements data from the components requirements database and sends the MR information to the components requirements database. An AI/ML algorithm analyzes the input information from the dataset and prepares an updated component dataset.

Inventors

• Ramesh Sethu
• Rami Ismail Debouk
• Paolo Giusto
• Azeem Sarwar

Assignees

• GM Global Technology Operations LLC

Dates

Publication Date

20260505

Application Date

20221118

Claims (17)

1. 1 . A system for (continuous integration continuous deployment) CI/CD of (artificial intelligence/machine learning) AI/ML based components, comprising: a cloud infrastructure which receives multiple new requirements from a vehicle designer or a developer, with the new requirements adapted for artificial intelligence/machine learning (AI/ML) based components of a vehicle; a dataset; a metamorphic relations (MR) module which receives input information from the dataset and sends MR information to the dataset; a components requirements database which includes the new requirements in addition to existing requirements for the AI/ML based components; the MR module also receiving components requirements data from the components requirements database and sending the MR information to the components requirements database, wherein the MR information defines a variation due to inputs such as a small change in input having a small change in output; and an AI/ML algorithm which analyzes the input information from the dataset and prepares an updated component dataset, wherein the updated component dataset is transmitted as an over-the-air (OTA) update to a vehicle requirements database for deployment to AI/ML based components of a vehicle.
2. 2 . The system for CI/CD of AI/ML based components of claim 1 , further including multiple test cases data also received from and sent to the MR module.
3. 3 . The system for CI/CD of AI/ML based components of claim 2 , including a vehicle requirements database having multiple edge cases from multiple vehicles.
4. 4 . The system for CI/CD of AI/ML based components of claim 1 , including a cloud-based architecture supporting CI/CD of the AI/ML based components.
5. 5 . The system for CI/CD of AI/ML based components of claim 4 , including a requirement capturing module (RCM) receiving the new requirements as input from the vehicle designer or the developer.
6. 6 . The system for CI/CD of AI/ML based components of claim 5 , including an asset refinement engine (ARE) which receives refined requirements from the RCM.
7. 7 . The system for CI/CD of AI/ML based components of claim 6 , including an edge case capturing component (ECC) which receives requirements data from a vehicle requirements database and generates inputs to the AI/ML based components for situations when the AI/ML based components may perform poorly due to images in low light conditions, images with overlapping traffic participants, or images with long shadows or degraded lane markers, with the inputs to the ECC from the vehicle requirements database being generated directly from real-life scenarios.
8. 8 . The system for CI/CD of AI/ML based components of claim 7 , including an AI/ML development platform (ADP) receiving an output of the ARE and from the dataset, the ADP being used for all AI/ML based components; and wherein an output from the ADP is forwarded to the AI/ML algorithms.
9. 9 . The system for CI/CD of AI/ML based components of claim 6 , wherein the ARE: employs processes and methods for validation, generation, and refinement of requirements, datasets and test cases; provides a process and method for simultaneous refinement of requirements, datasets and test cases; provides representation of requirements as metamorphic relations (MRs), and extraction of MRs from the requirements, datasets and test cases; and converts MRs as satisfiability modulo theory (SMT) formulae; and wherein an output of the ARE is forwarded to the dataset.
10. 10 . The system for CI/CD of AI/ML based components of claim 1 , including a vehicle requirements database, wherein the updated component dataset is forwarded as an over-the-air (OTA) update to the vehicle requirements database.
11. 11 . A method for continuous integration continuous deployment (CI/CD) of artificial intelligence/machine learning (AI/ML) based components using a cloud-based architecture, comprising: performing CI/CD of AI/ML component development; entering input data from a vehicle requirements database into an edge case capturing component (ECC) and applying the ECC to generate inputs to the AI/ML component development; inputting design data as design requirements from a vehicle designer or developer into a requirements capturing module (RCM); providing the design requirements from the RCM to an asset refinement engine (ARE) to validate, generate, and refine the design requirements; analyzing data retrieved from a dataset and preparing an updated component dataset using an AI/ML algorithm; transmitting the updated component dataset as an over-the-air (OTA) update to a vehicle requirements database in a vehicle; and deploying the updated component dataset to AI/ML based components of the vehicle.
12. 12 . The method of claim 11 , further including providing representation of the design requirements as metamorphic relations (MRs) using the ARE.
13. 13 . The method of claim 12 , further including extracting the MRs from the design requirements using the ARE.
14. 14 . The method of claim 13 , further including converting the MRs as satisfiability modulo theory (SMT) formulae using the ARE.
15. 15 . The method of claim 11 , further including operating a metamorphic relations module receiving data from and sending data to a components requirements database to check consistency of behaviors across variations and to generate new test cases.
16. 16 . The method of claim 11 , further including sending an output of the ARE and data from the dataset to an AI/ML development platform (ADP).
17. 17 . The method of claim 11 , further including operating the ECC to perform a pre-analysis and filter the inputs to the AI/ML based components.

Description

INTRODUCTION The present disclosure relates to artificial intelligence/machine learning based components used in autonomous vehicles. Artificial intelligence/machine learning (AI/ML) based components perform many functionalities in autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) and their verification and validation is important for the correctness of these functionalities. The specification of the behavior of AI/ML based components describe complex human based concepts such as lanes, pedestrians, traffic signals and the like. AI/ML based components are typically developed starting from a collected sample dataset of examples which ultimately defines the final component behaviors. AI/ML based components are eventually implemented as software (SW) components mapped to hardware (HW) resources and are used across multiple different inputs, environments, and platforms. Metamorphic relations (MRs) capture relationships including differences between the components. Many examples of MRs exist: a simplest one defining a variation due to inputs such as a small change in input having a small change in the outputs, for example, |F(x+d)−F(d)|<e. While current AI/ML based components and product development processes achieve their intended functionalities, there is a need for a new and improved system and method for development of AI/ML based components. SUMMARY According to several aspects, a system for continuous integration/continuous deployment (CI/CD) of artificial intelligence/machine learning (AI/ML) based components incorporated in a computational cloud infrastructure includes a requirements database system receiving new requirements from a vehicle designer or developer, with the new requirements adapted for AI/ML based components of a vehicle, the system including a dataset and a metamorphic relations (MR) module receiving input information from the dataset and sending MR information to the dataset. The components requirements database includes the new requirements in addition to existing requirements for the AI/ML based components. The MR module also receives components requirements in the form of metamorphic relations from the components requirements database and, in turn, sends new MR information to the components requirements database. A machine learning based system analyzes the input information from the dataset and prepares an updated component dataset. In another aspect of the present disclosure, multiple test cases data is also received from and sent to the MR module. In another aspect of the present disclosure, a vehicle requirements database includes multiple edge cases from multiple vehicles, the edge cases defining challenging scenarios in which performance of AI/ML components are often poor. In another aspect of the present disclosure, the computational cloud infrastructure supports continuous integration/continuous deployment (CI/CD) of the AI/ML components. In another aspect of the present disclosure, a requirements capturing module (RCM) receives the new requirements as input from the vehicle designer or the developer. In another aspect of the present disclosure, an asset refinement engine (ARE) receiving refined requirements from the RCM. In another aspect of the present disclosure, an edge case capturing component (ECC) receives requirements data from the vehicle requirements database and generates inputs to the AI/ML based components for situations when the AI/ML components may perform poorly due to images in low light conditions, images with overlapping traffic participants, or images with long shadows or degraded lane markers, with the inputs to the ECC from the vehicle requirements database being generated directly from real-life scenarios. In another aspect of the present disclosure, an AI/ML development platform (ADP) receives an output of the ARE and from the dataset, the ADP being used for all AI/ML based components; and wherein an output from the ADP is forwarded to the AI/ML algorithms. In another aspect of the present disclosure, the ARE employs processes and methods for validation, generation, and refinement of requirements, datasets and test cases, provides a process and method for simultaneous refinement of requirements, datasets and test cases, provides representation of requirements as metamorphic relations (MRs), and extraction of MRs from the requirements, datasets and test cases, and converts MRs as satisfiability modulo theory (SMT) formulae; wherein an output of the ARE is forwarded to the dataset. In another aspect of the present disclosure, the updated component requirements are forwarded as an over-the-air (OTA) update to the vehicle requirements database. According to several aspects, a method for continuous integration/continuous deployment (CI/CD) of artificial intelligence/machine learning (AI/ML) based components using a cloud-based architecture comprises: performing CI/CD of AI/ML based components; entering input data from a vehicle requirem