US-20260127098-A1 - USING MACHINE LEARNING TO PREDICT TESTS ASSOCIATED WITH APPLICATION PROGRAMMING INTERFACE UPDATES

US20260127098A1US 20260127098 A1US20260127098 A1US 20260127098A1US-20260127098-A1

Abstract

Systems and methods for using machine learning to predict tests for testing application programming interface updates. The system receives a prediction request to predict one or more tests for testing a candidate update to an application programming interface, wherein the prediction request comprises an indication of the application programming interface. The system may determine, using the indication of the application programming interface, a plurality of parameters associated with the application programming interface and obtain, based on the plurality of parameters using a machine learning model a prediction of a plurality of tests to be executed for the candidate update to the application programming interface. The system may determine, a first subset of the plurality of tests enabled to be executed without user input and cause the first subset of the plurality of tests to be executed without the user input.

Inventors

Vidyalakshmi PATHAI RAMAKRISHNAN
Padmapriya NARASIMHAN
Premkumar RAJAGOPALAN
Miriam Silver

Assignees

CITIBANK, N.A.

Dates

Publication Date: 20260507
Application Date: 20241107

Claims (20)

1 . A system for using machine learning to predict tests for testing application programming interface updates, the system comprising: one or more processors; and one or more memories configured to store instructions that when executed by the one or more processors perform operations comprising: receiving a prediction request to predict one or more tests for testing a candidate update to an application programming interface, wherein the prediction request comprises at least one identifier of the application programming interface; determining, using the at least one identifier of the application programming interface, a plurality of parameters associated with the application programming interface, wherein the plurality of parameters is obtained from one or more parameter sources; retrieving, by querying a retrieval-augmented generation (RAG) system using at least one of the plurality of parameters, data corresponding to one or more potential dependencies of the application programming interface; inputting (a) the plurality of parameters and (b) the data corresponding to the one or more potential dependencies into a machine learning model to obtain a prediction of a plurality of tests to be executed for the candidate update to the application programming interface, wherein the machine learning model has been trained to predict, based on the plurality of parameters embedded into an embedding space of the machine learning model, the prediction of the plurality of tests; receiving, from the machine learning model, the plurality of tests as an output; determining, based on run-time parameters of each test of the plurality of tests, a first subset of the plurality of tests enabled to be executed without user input and a second subset of the plurality of tests that are not enabled to be executed without the user input, wherein each run-time parameter is related to execution of a corresponding test of the plurality of tests; and executing the first subset of the plurality of tests without the user input.
2 . The system of claim 1 , wherein the instructions for determining the plurality of parameters associated with the application programming interface further cause the one or more processors to perform operations comprising: receiving computer code of the application programming interface; generating, using an embedding model trained to embed the computer code into the embedding space of the machine learning model, an embedding representing the application programming interface; and adding the embedding as a parameter of the plurality of parameters.
3 . The system of claim 2 , wherein the instructions for inputting the plurality of parameters into the machine learning model to obtain the prediction of the plurality of tests further cause the one or more processors to input the embedding into the machine learning model.
4 . The system of claim 1 , wherein the instructions further cause the one or more processors to perform operations comprising: retrieving, from a development database, a first training dataset comprising a first plurality of tests associated with a plurality of application programming interfaces, and, from a repository, a second training dataset comprising a second plurality of tests associated with the plurality of application programming interfaces; generating, using an embedding model trained to embed computer code into the embedding space of the machine learning model, a plurality of embeddings representing the plurality of application programming interfaces; and training the machine learning model using a training dataset comprising the first training dataset, the second training dataset, and the plurality of embeddings.
5 . The system of claim 1 , wherein the instructions further cause the one or more processors to perform operations comprising: retrieving, for a plurality of application programming interfaces, a corresponding plurality of tests, wherein each corresponding plurality of tests were performed on a corresponding application programming interface; generating a first portion of a prompt for the machine learning model that includes a command to use the plurality of application programming interfaces and each corresponding plurality of tests to inform a response from the machine learning model, wherein the machine learning model is a large language model; generating a second portion of the prompt for the machine learning model based on the application programming interface; and inputting the prompt into the large language model.
6 . The system of claim 1 , wherein the instructions further cause the one or more processors to perform operations comprising: receiving a message indicating that a test is missing from the prediction of the plurality of tests; generating an updated plurality of tests based on adding the test that is missing to the plurality of tests; generating an updated embedding from the application programming interface; and inputting the updated embedding and the updated plurality of tests into a training routine of the machine learning model to improve the machine learning model.
7 . A method for using machine learning to predict tests for testing application programming interface updates, the method comprising: receiving a prediction request to predict one or more tests for testing a candidate update to an application programming interface, wherein the prediction request comprises at least one identifier of the application programming interface; determining, using the at least one identifier of the application programming interface, a plurality of parameters associated with the application programming interface; retrieving, by querying a retrieval-augmented generation (RAG) system using at least one of the plurality of parameters, data corresponding to one or more potential dependencies of the application programming interface; inputting (a) the plurality of parameters and (b) the data corresponding to the one or more potential dependencies into a machine learning model to obtain a prediction of a plurality of tests to be executed for the candidate update to the application programming interface, wherein the machine learning model has been trained to predict, based on the plurality of parameters, the prediction of the plurality of tests; receiving, from the machine learning model, the plurality of tests as an output; determining, based on run-time parameters of each test of the plurality of tests, a first subset of the plurality of tests enabled to be executed without user input, wherein each run-time parameter is related to execution of a corresponding test of the plurality of tests; determining, based on run-time parameters of each test, a second subset of the plurality of tests that are not enabled to be executed without user input; and executing the first subset of the plurality of tests without the user input and notifying one or more users of the second subset of the plurality of tests.
8 . The method of claim 7 , wherein the plurality of parameters is embedded into an embedding space of the machine learning model using an embedding model.
9 . The method of claim 7 , wherein determining the plurality of parameters associated with the application programming interface further comprises: receiving computer code of the application programming interface; generating, using an embedding model trained to embed the computer code into an embedding space of the machine learning model, an embedding representing the application programming interface; and adding the embedding as a parameter of the plurality of parameters.
10 . The method of claim 9 , wherein inputting the plurality of parameters into the machine learning model to obtain the prediction of the plurality of tests further comprises inputting the embedding of the application programming interface into the machine learning model.
11 . The method of claim 7 , further comprising: retrieving, from a development database, a first training dataset comprising a first plurality of tests associated with a plurality of application programming interfaces, and, from a repository, a second training dataset comprising a second plurality of tests associated with the plurality of application programming interfaces; generating, using an embedding model trained to embed computer code into an embedding space of the machine learning model, a plurality of embeddings representing the plurality of application programming interfaces; and training the machine learning model using a training dataset comprising the first training dataset, the second training dataset, and the plurality of embeddings.
12 . The method of claim 7 , further comprising: retrieving, for a plurality of application programming interfaces, a corresponding plurality of tests, wherein each corresponding plurality of tests were performed on a corresponding application programming interface; generating a first portion of a prompt for the machine learning model that includes a command to use the plurality of application programming interfaces and each corresponding plurality of tests to inform a response from the machine learning model, wherein the machine learning model is a large language model; generating a second portion of the prompt for the machine learning model based on the application programming interface; and inputting the prompt into the large language model.
13 . The method of claim 7 , further comprising: receiving a message indicating that a test is missing from the prediction of the plurality of tests; generating an updated plurality of tests based on adding the test that is missing to the plurality of tests; generating an updated embedding from the application programming interface; and inputting the updated embedding and the updated plurality of tests into a training routine of the machine learning model to improve the machine learning model.
14 . One or more non-transitory, computer-readable media for providing user interfaces using artificial intelligence, storing instructions thereon that cause one or more processors to perform operations comprising: receiving a prediction request to predict one or more tests for testing a candidate update to an application programming interface, wherein the prediction request comprises at least one identifier of the application programming interface; determining, using the at least one identifier of the application programming interface, a plurality of parameters associated with the application programming interface; retrieving, by querying a retrieval-augmented generation (RAG) system using at least one of the plurality of parameters, data corresponding to one or more potential dependencies of the application programming interface; obtaining, based on (a) the plurality of parameters and (b) the data corresponding to the one or more potential dependencies and using a machine learning model, a prediction of a plurality of tests to be executed for the candidate update to the application programming interface; receiving, from the machine learning model, the plurality of tests as an output; determining, based on run-time parameters of each test of the plurality of tests, a first subset of the plurality of tests enabled to be executed without user input, wherein each run-time parameter is related to execution of a corresponding test of the plurality of tests; determining, based on run-time parameters of each test, a second subset of the plurality of tests that are not enabled to be executed without user input; and executing the first subset of the plurality of tests without the user input and notifying one or more users of the second subset of the plurality of tests.
15 . The one or more non-transitory, computer-readable media of claim 14 , wherein instructions further cause the one or more processors to embed the plurality of parameters into an embedding space of the machine learning model using an embedding model.
16 . The one or more non-transitory, computer-readable media of claim 14 , wherein the instructions for determining the plurality of parameters associated with the application programming interface further cause the one or more processors to perform operations comprising: receiving computer code of the application programming interface; generating, using an embedding model trained to embed the computer code into an embedding space of the machine learning model, an embedding representing the application programming interface; and adding the embedding as a parameter of the plurality of parameters.
17 . The one or more non-transitory, computer-readable media of claim 16 , wherein the instructions for inputting the plurality of parameters into the machine learning model to obtain the prediction of the plurality of tests further cause the one or more processors to input the embedding of the application programming interface into the machine learning model.
18 . The one or more non-transitory, computer-readable media of claim 14 , wherein the instructions further cause the one or more processors to perform operations comprising: retrieving, from a development database, a first training dataset comprising a first plurality of tests associated with a plurality of application programming interfaces, and, from a repository, a second training dataset comprising a second plurality of tests associated with the plurality of application programming interfaces; generating, using an embedding model trained to embed computer code into an embedding space of the machine learning model, a plurality of embeddings representing the plurality of application programming interfaces; and training the machine learning model using a training dataset comprising the first training dataset, the second training dataset, and the plurality of embeddings.
19 . The one or more non-transitory, computer-readable media of claim 14 , wherein the instructions further cause the one or more processors to perform operations comprising: retrieving, for a plurality of application programming interfaces, a corresponding plurality of tests, wherein each corresponding plurality of tests were performed on a corresponding application programming interface; generating a first portion of a prompt for the machine learning model that includes a command to use the plurality of application programming interfaces and each corresponding plurality of tests to inform a response from the machine learning model, wherein the machine learning model is a large language model; generating a second portion of the prompt for the machine learning model based on the application programming interface; and inputting the prompt into the large language model.
20 . The one or more non-transitory, computer-readable media of claim 14 , wherein the instructions further cause the one or more processors to perform operations comprising: receiving a message indicating that a test is missing from the prediction of the plurality of tests; generating an updated plurality of tests based on adding the test that is missing to the plurality of tests; generating an updated embedding from the application programming interface; and inputting the updated embedding and the updated plurality of tests into a training routine of the machine learning model to improve the machine learning model.

Description

BACKGROUND Software development and testing has been on the forefront of computer science for many years. Engineers have been updating computer code for various reasons including new features, bug fixes and for other suitable reasons. In many instances this computer code takes a form of an application programming interface that can be invoked to perform various tasks. In recent years, engineers have been developing more and more complex application programming interfaces (“APIs”) and it has become difficult to update those APIs and test the updates without breaking any computer code that uses those application programming interfaces. In addition, in recent years the use of artificial intelligence, including but not limited to machine learning, deep learning, etc. (referred to collectively herein as artificial intelligence), has exponentially increased. Broadly described, artificial intelligence refers to a wide-ranging branch of computer science concerned with building smart machines capable of performing tasks that typically require human intelligence. Key benefits of artificial intelligence are its ability to process data, find underlying patterns, and/or perform real-time determinations. Thus, it may be desirable to use artificial intelligence (e.g., machine learning) to aid in properly updating APIs. SUMMARY Accordingly, systems and methods are described herein for using artificial intelligence such as machine learning to predict tests for testing application programming interface updates. An API testing system may be used to perform operations described herein. For example, when developers generate updates to user interfaces, these developers may be required to test the updated APIs. However, it may be difficult to ascertain which tests should be run and who should run those tests. In some instances, the tests may be stored in two different databases and may be retrieved and displayed to a user for execution. For example, a developer database may include tests recommended by developers based on developers'knowledge of API usage. A second database may be queried that may store other candidate tests. For example, if the same API was updated in the past, those tests may be stored with the API's identifier. Thus, the API testing system may retrieve those tests and determine that those tests should be added to the testing of the API. In this case, any missed tests would be added to the testing schedule. In some embodiments, the API testing system may use machine learning to determine which tests should be performed on the updated API. In particular, the API testing system may receive a prediction request to predict one or more tests for testing a candidate update to an application programming interface. The prediction request may include an indication of the application programming interface. For example, a developer or a team of developers may develop an update to a particular API and may submit the update to be tested. Thus, the API testing system may receive the prediction request. When the API testing system receives the prediction request, the API testing system may determine parameters for performing the prediction. In particular, the API testing system may determine, using the indication of the application programming interface, a plurality of parameters associated with the application programming interface. In some embodiments, the parameters may be metadata associated with the API (e.g., variables, sub-routines, jurisdictions of where the API is used, and others). However, in some embodiments, the parameters may be one or more embeddings of the API code. For example, the API testing system may embed the code of the API into an embedding space of a machine learning model. In another example, the API testing system may vectorize the code of the API into a vector space of the machine learning model. For example, the embedding model may be trained by retrieving, e.g., from a development database, a first training dataset that includes a first plurality of tests associated with a plurality of application programming interfaces, and, from a repository, a second training dataset that includes a second plurality of tests associated with the plurality of application programming interfaces. Using an embedding model trained to embed computer code into the embedding space of the machine learning model, a plurality of embeddings can be generated. In some examples, the plurality of embeddings may represent the plurality of application programming interfaces. The machine learning model may then be trained using a training dataset that includes the first training dataset, the second training dataset, and the plurality of embeddings. The API testing system may then use the machine learning model to predict which tests should be used to test the updated API. In particular, the API testing system may input the plurality of parameters into a machine learning model to obtain a prediction of a plurality of tests to be e