US-12626151-B2 - Method and system for testing a classification machine learning (ML) model of a tenant of a service provider, in a cloud-based environment

US12626151B2US 12626151 B2US12626151 B2US 12626151B2US-12626151-B2

Abstract

A computerized-method for testing a classification ML model of a tenant of a service provider, in a cloud-based environment. The computerized-method includes: (i) receiving an object of a classification ML model for testing from the tenant; (ii) executing an API with the received object of the classification ML model; (iii) identifying one or more tenants of the service provider based on an activity type and preconfigured characteristics by the executed API; (iv) performing an evaluation of the object of the classification ML model by operating the API on each retrieved dataset of the one or more tenants of the service provider to evaluate the object of the classification ML model and store score-results; and (v) calculating an average of the stored score-results to yield a performance-score of the classification ML model. When the performance-score is above a predefined performance-score deploying the classification ML model in a system of the tenant.

Inventors

Sunny THOLAR
Ori Snir
Amir Shachar

Assignees

ACTIMIZE LTD.

Dates

Publication Date: 20260512
Application Date: 20230222

Claims (12)

1 . A computerized-method for testing a classification Machine Learning (ML) model of a tenant of a service provider, in a cloud-based environment, said computerized-method comprising: (i) receiving an object of a classification ML model for testing, from the tenant; (ii) executing an Application Programming Interface (API) with the received object of the classification ML model; (iii) identifying one or more tenants of the service provider based on an activity type and preconfigured characteristics by the executed API, wherein the tenants are financial Institutions (FI)s, and wherein the preconfigured characteristics are selected from at least one of: (i) fraud rate; (iii) number of transactions in a preconfigured period; (iii) unique payee; (iv) average daily transaction; (v) average weekly transaction; and (vi) number of clean transaction divided by number of fraud transaction, (iv) performing an evaluation of the object of the classification ML model by operating the API on each retrieved dataset of the one or more tenants of the service provider to evaluate the object of the classification ML model and store score-results, wherein each dataset is having similar attributes as the received object of classification ML model; and (v) calculating an average of the stored score-results to yield a performance-score of the classification ML model, wherein, when the performance-score is above a predefined performance-score deploying the classification ML model in a system of the tenant of the service provider.
2 . The computerized-method of claim 1 , wherein the identified one or more tenants share similar traits to the tenant that is having the classification ML model tested.
3 . The computerized-method of claim 1 , wherein the evaluation of the object of the classification ML model is operated on one or more datasets of tenants of the service provider.
4 . The computerized-method of claim 1 , wherein the evaluation of the object of the classification ML model is by at least one parameters of: (i) feature list; (ii) list of alert rate; and (iii) month.
5 . The computerized-method of claim 1 , wherein the evaluation is performed according to detection rate and value detection rate for a list of alert rates provided by the tenant.
6 . The computerized-method of claim 1 , wherein data of each retrieved dataset of the one or more tenants of the service provider is uploaded to a cloud object storage from one or more databases in each identified tenant system.
7 . The computerized-method of claim 1 , wherein the one or more databases are selected from at least one of: (i) customer database; (ii) recent data database; and (iii) behavioral profiles database.
8 . The computerized-method of claim 1 , wherein the system of the tenant of the service provider that the classification ML model is deployed in is an Integrated Fraud Management (IFM) system for automatically scoring financial transactions by the operation of the deployed classification ML model.
9 . The computerized-method of claim 8 , wherein based on a score of a financial transaction the financial transaction is allowed or declined or delayed until an operator action is taken regarding the transaction.
10 . The computerized-method of claim 1 , wherein the activity is selected from at least one of: (i) Person to Person (P2P) transfer; and (ii) Automated Clearing House (ACH) transfer; (iii) checks deposit; (iv) non wire transfer; and (v) wire transfer.
11 . A computerized-system for testing a classification Machine Learning (ML) model of a tenant of a service provider, in a cloud-based environment, said computerized-system comprising: one or more processors, said one or more processors are configured to: (i) receive an object of a classification ML model for testing, from the tenant; (ii) execute an Application Programming Interface (API) with the received object of the classification ML model; (iii) identify one or more tenants of the service provider based on an activity type and preconfigured characteristics by the executed API, wherein the tenants are financial Institutions (FI)s, and wherein the preconfigured characteristics are selected from at least one of: (i) fraud rate; (iii) number of transactions in a preconfigured period; (iii) unique payee; (iv) average daily transaction; (v) average weekly transaction; and (vi) number of clean transaction divided by number of fraud transaction, (iv) perform an evaluation of the object of the classification ML model by operating the API on each retrieved dataset of the one or more tenants of the service provider to evaluate the object of the classification ML model and store score-results, wherein each dataset is having similar attributes as the received object of classification ML model; and (v) calculate an average of the stored score-results to yield a performance-score of the classification ML model, wherein, when the performance-score is above a predefined performance-score deploying the classification ML model in a system of the tenant of the service provider.
12 . A computerized-method for training and testing a classification Machine Learning (ML) model of a tenant of a service provider, in a cloud-based environment, said computerized-method comprising: (i) performing Hyper Parameter Optimization (HPO) on a financial crime and compliance ecosystem platform to yield performance-results, wherein the financial crime and compliance ecosystem platform communicates with an HPO Application Programming Interface (API); (ii) evaluating the yielded performance-results to return a preconfigured number of parameters having top performance-results; (iii) using the preconfigured number of top performing hyper parameters to create a classification ML model of each parameter of the preconfigured number of parameters having top performance-results; and (iv) testing each created classification ML model by: a. executing an API with the received object of the classification ML model; b. identifying one or more tenants of the service provider based on an activity type and preconfigured characteristics by the executed API, wherein the tenants are financial Institutions (FI)s, and wherein the preconfigured characteristics are selected from at least one of: (i) fraud rate; (iii) number of transactions in a preconfigured period; (iii) unique payee; (iv) average daily transaction; (v) average weekly transaction; and (vi) number of clean transaction divided by number of fraud transaction, c. performing an evaluation of the object of the classification ML model by operating the API on each retrieved dataset of the one or more tenants of the service provider to evaluate the object of the classification ML model and store score-results, wherein each dataset is having similar attributes as the received object of classification ML model; and d. calculating an average of the stored score-results to yield a performance-score of the classification ML model, wherein, when the performance-score is above a predefined performance-score deploying the classification ML model in a system of the tenant of the service provider.

Description

TECHNICAL FIELD The present disclosure relates to the field of Machine Learning (ML) based systems and more specifically to Machine Learning (ML) model training, evaluation and selection in a cloud-based environment. BACKGROUND An Artificial Intelligence (AI)-based system is a computer system which is able to perform tasks that ordinarily require human intelligence. Many of these AI-based systems are powered by ML models, some of them are powered by deep learning models and some of them are powered by rules-based ML models. Financial institutions often lack high-quality fraud data to test these ML models with. For example, low fraud counts, as compared to legit data, may challenge the creation and the testing of a robust ML model. Due to the low fraud counts financial, financial institutions may use most of the fraud transaction for training and as a result very few numbers of frauds are left for the testing stage of the ML model that has been created. Furthermore, there are situations where the financial institutions may have sufficient data to train the ML model but no data to perform Hyper Parameter Optimization (HPO) as part of the training phase and consequently the client trains the ML model with a default set of parameters instead of optimized hyper parameters which results in a less robust ML model. As a result of the described lacuna, financial institutions are hesitant to deploy these ML models into production as they are concerned with unpredicted data patterns which may be overlooked by the ML model and as a result expose the financial institutions to increased losses. Accordingly, there is a need for a technical solution that will provide financial institutions a mechanism that enables testing of the fraud detection ML models on data assets and further provides a good validation of performance of these tested fraud detection models as well as enables HPO as part of the training phase. There is a need for a system and method for testing a classification Machine Learning (ML) model of a tenant of a service provider, in a cloud-based environment and system and method for training and testing a classification ML model of a tenant of a service provider, in the cloud-based environment. SUMMARY There is thus provided, in accordance with some embodiments of the present disclosure, a computerized-method for testing a classification Machine Learning (ML) model of a tenant of a service provider, in a cloud-based environment. In accordance with some embodiments of the present disclosure, the computerized-method includes: (i) receiving an object of a classification ML model for testing, from the tenant; (ii) executing an Application Programming Interface (API) with the received object of the classification ML model; (iii) identifying one or more tenants of the service provider based on an activity type and preconfigured characteristics by the executed API; (vi) performing an evaluation of the object of the classification ML model by operating the API on each retrieved dataset of the one or more tenants of the service provider to evaluate the object of the classification ML model and store score-results, each dataset is having similar attributes as the received object of classification ML model; and (v) calculating an average of the stored score-results to yield a performance-score of the classification ML model. When the performance-score is above a predefined performance-score deploying the classification ML model in a system of the tenant of the service provider. Furthermore, in accordance with some embodiments of the present disclosure, the tenants may be Financial Institutions (FI)s. Furthermore, in accordance with some embodiments of the present disclosure, the preconfigured characteristics may be selected from at least one of: (i) fraud rate; (iii) number of transactions in a preconfigured period; (iii) unique payee; (iv) average daily transaction; (v) average weekly transaction; and (vi) number of clean transaction divided by number of fraud transaction. Furthermore, in accordance with some embodiments of the present disclosure, the identified one or more tenants may share similar traits to the tenant that is having the classification ML model tested. Furthermore, in accordance with some embodiments of the present disclosure, the evaluation of the object of the classification ML model may be operated on one or more datasets of tenants of the service provider. Furthermore, in accordance with some embodiments of the present disclosure, the evaluation of the object of the classification ML model may be by at least one parameters of: (i) feature list; (ii) list of alert rate; and (iii) month. Furthermore, in accordance with some embodiments of the present disclosure, the evaluation may be performed according to detection rate and value detection rate for a list of alert rates provided by the tenant. Furthermore, in accordance with some embodiments of the present disclosure, data of each retrieved dataset