US-12620025-B2 - Machine learning model selection for accounts receivable predictions

US12620025B2US 12620025 B2US12620025 B2US 12620025B2US-12620025-B2

Abstract

Embodiments predict a target variable for accounts receivable using a machine learning model. For a first customer, embodiments receive a plurality of trained ML models corresponding to the target variable, the plurality of trained ML models trained using the historical data and comprising a first trained model having no grace period for the target variable and two or more grace period trained models, each grace period trained model having different grace periods for the target variable. Embodiments determine a Matthews' Correlation Coefficient (“MCC”) for the first trained model. When the MCC for the first trained model is low, embodiments determine the MCC for each of the grace period trained models, and when one or more MCCs for each of the grace period trained models is higher than the MCC for the first trained model, embodiments select the corresponding grace period trained model having a highest MCC.

Inventors

Vikas Agrawal
Krishnan Ramanathan
Praneeth Medhatithi SHISHTLA
Jagdish Chand

Assignees

ORACLE INTERNATIONAL CORPORATION

Dates

Publication Date: 20260505
Application Date: 20230906

Claims (20)

1 . A method of optimizing a predicting of a target variable using a system comprising a plurality of different machine learning (ML) models in a cloud based analytics system for a tenant of the cloud based analytics system, the method comprising: receiving historical data corresponding to a plurality of transactions corresponding to a plurality of customers of the tenant, the historical data comprising, for each of the transactions, the target variable; for a first customer, receiving a plurality of trained ML models corresponding to the target variable, the plurality of trained ML models trained using the historical data and comprising a first trained model having no grace period for the target variable and two or more different grace period trained models, each grace period trained model having different grace periods for the target variable; determining a Matthews' Correlation Coefficient (MCC) for the first trained model; when the MCC for the first trained model is low, determining the MCC for each of the grace period trained models, and when one or more MCCs for each of the grace period trained models is higher than the MCC for the first trained model and exceed a threshold MCC, selecting a corresponding grace period trained model having a highest MCC; and when the first trained model or the selected grace period trained model has a high MCC, deploying the first trained model or the selected grace period trained model to predict the target variable; wherein receiving the historical data comprises retrieving the historical data as a dataset from a transactional database corresponding to the tenant in response to an activation plan, the activation plan: extracting the historical data to a data staging area corresponding to the tenant; transforming the historical data into a data warehouse format by a data transformation layer; and loading the transformed historical data into a data warehouse for the cloud based analytics system corresponding to the tenant.
2 . The method of claim 1 , further comprising: when the MCC for each of the grace period trained models is lower than the MCC for the first trained model, not deploying the first trained model or the selected grace period trained model to predict the target variable.
3 . The method of claim 1 , further comprising: when the first trained model or the selected grace period trained model has a mid range MCC, segmenting each of the transactions for the first customer, the segmenting comprising determining a measure of variability of the target variable for each transaction and, based on the measure of variability, classifying each transaction as having a low variation, a medium variation, or a high variation.
4 . The method of claim 3 , wherein the low MCC is approximately less than 0.3, the mid range MCC is approximately 0.3-0.6, and the high MCC is approximately greater than 0.6.
5 . The method of claim 4 , wherein: MCC = TP × TN - FP × FN ( TP + FP ) ⁢ ( TP + FN ) ⁢ ( TN + FP ) ⁢ ( TN + FN ) where TP is a number of true positives, TN is a number of true negatives, FP is a number of false positives and FN is a number of false negatives.
6 . The method of claim 3 , wherein determining the measure of variability for the first customer comprises using a median based coefficient of variation.
7 . The method of claim 6 , wherein the median based coefficient of variation comprises: median ⁢ of ⁢ ❘ "\[LeftBracketingBar]" X i - median ( X ) ❘ "\[RightBracketingBar]" median ( X ) where X i is a value of the target variable for each transaction i of the first customer and median (X) is the median of the target variables from all transactions of the first customer.
8 . The method claim 1 , wherein the target variable is a number of days that a payment is delayed after a payment due date for each transaction and the transforming the historical data into the data warehouse format comprises at least one of dimension generation, fact generation or aggregate generation.
9 . The method of claim 1 , further comprising: segmenting each of the customers based on the historical data corresponding to each of the customers, the segmenting comprising determining a measure of variability of the target variable for each customer and, based on the measure of variability, classifying each customer as having a low variation, a medium variation, or a high variation; for each low variation customer, creating the first trained model; and for each medium variation customer, creating the first trained model and creating the two or more grace period trained models.
10 . A non-transitory computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the processors to optimize a prediction of a target variable using a system comprising a plurality of different machine learning (ML) models in a cloud based analytics system for a tenant of the cloud based analytics system, the predicting comprising: receiving historical data corresponding to a plurality of transactions corresponding to a plurality of customers of the tenant, the historical data comprising, for each of the transactions, the target variable; for a first customer, receiving a plurality of trained ML models corresponding to the target variable, the plurality of trained ML models trained using the historical data and comprising a first trained model having no grace period for the target variable and two or more grace period trained models, each grace period trained model having different grace periods for the target variable; determining a Matthews' Correlation Coefficient (MCC) for the first trained model; when the MCC for the first trained model is low, determining the MCC for each of the grace period trained models, and when one or more MCCs for each of the grace period trained models is higher than the MCC for the first trained model, selecting a corresponding grace period trained model having a highest MCC; and when the first trained model or the selected grace period trained model has a high MCC, deploying the first trained model or the selected grace period trained model to predict the target variable; wherein receiving the historical data comprises retrieving the historical data as a dataset from a transactional database corresponding to the tenant in response to an activation plan, the activation plan: extracting the historical data to a data staging area corresponding to the tenant; transforming the historical data into a data warehouse format by a data transformation layer; and loading the transformed historical data into a data warehouse for the cloud based analytics system corresponding to the tenant.
11 . The computer readable medium of claim 10 , the predicting further comprising: when the MCC for each of the grace period trained models is lower than the MCC for the first trained model, not deploying the first trained model or the selected grace period trained model to predict the target variable.
12 . The computer readable medium of claim 10 , the predicting further comprising: when the first trained model or the selected grace period trained model has a mid range MCC, segmenting each of the transactions for the first customer, the segmenting comprising determining a measure of variability of the target variable for each transaction and, based on the measure of variability, classifying each transaction as having a low variation, a medium variation, or a high variation.
13 . The computer readable medium of claim 12 , wherein the low MCC is approximately less than 0.3, the mid range MCC is approximately 0.3-0.6, and the high MCC is approximately greater than 0.6.
14 . The computer readable medium of claim 13 , wherein: MCC = TP × TN - FP × FN ( TP + FP ) ⁢ ( TP + FN ) ⁢ ( TN + FP ) ⁢ ( TN + FN ) where TP is a number of true positives, TN is a number of true negatives, FP is a number of false positives and FN is a number of false negatives.
15 . The computer readable medium of claim 12 , wherein determining the measure of variability for the first customer comprises using a median based coefficient of variation.
16 . The computer readable medium of claim 15 , wherein the median based coefficient of variation comprises: median ⁢ of ⁢ ❘ "\[LeftBracketingBar]" X i - median ( X ) ❘ "\[RightBracketingBar]" median ( X ) where X i is a value of the target variable for each transaction i of the first customer and median (X) is the median of the target variables from all transactions of the first customer.
17 . The computer readable medium claim 10 , wherein the target variable is a number of days that a payment is delayed after a payment due date for each transaction and the transforming the historical data into the data warehouse format comprises at least one of dimension generation, fact generation or aggregate generation.
18 . The computer readable medium of claim 10 , the predicting further comprising: segmenting each of the customers based on the historical data corresponding to each of the customers, the segmenting comprising determining a measure of variability of the target variable for each customer and, based on the measure of variability, classifying each customer as having a low variation, a medium variation, or a high variation; for each low variation customer, creating the first trained model; and for each medium variation customer, creating the first trained model and creating the two or more grace period trained models.
19 . A cloud based machine learning (ML) model analytics system for optimizing a predicting of a target variable for a tenant of the cloud based analytics system, the system comprising: a plurality of trained ML models comprising a first trained model having no grace period for the target variable and two or more grace period trained models, each grace period trained model having different grace periods for the target variable; one or more processors executing instructions and configured to: receive historical data corresponding to a plurality of transactions corresponding to a plurality of customers of the tenant, the historical data comprising, for each of the transactions, the target variable; for a first customer, using the a plurality of trained ML models corresponding to the target variable, the plurality of trained ML models trained using the historical data; determine a Matthews' Correlation Coefficient (MCC) for the first trained model; when the MCC for the first trained model is low, determine the MCC for each of the grace period trained models, and when one or more MCCs for each of the grace period trained models is higher than the MCC for the first trained model, select a corresponding grace period trained model having a highest MCC; and when the first trained model or the selected grace period trained model has a high MCC, deploy the first trained model or the selected grace period trained model to predict the target variable; wherein receiving the historical data comprises retrieving the historical data as a dataset from a transactional database corresponding to the tenant in response to an activation plan, the activation plan configured to: extracts the historical data to a data staging area corresponding to the tenant; transforms the historical data into a data warehouse format by a data transformation layer; and loads the transformed historical data into a data warehouse for the cloud based analytics system corresponding to the tenant.
20 . The system of claim 19 , further comprising: when the MCC for each of the grace period trained models is lower than the MCC for the first trained model, not deploying the first trained model or the selected grace period trained model to predict the target variable.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application Ser. No. 63/525,191 filed on Jul. 6, 2023, the disclosure of which is hereby incorporated by reference. FIELD One embodiment is directed generally to a machine learning model, and in particular to the generation and selection of machine learning models. BACKGROUND INFORMATION The process of generating or building a machine learning (“ML”) model includes multiple steps. The steps include gathering a suitable dataset for training the model and preprocessing the data by performing tasks such as cleaning, normalizing, and transforming it to a suitable format for training. Then the dataset is divided or split into two or three parts: the training set, validation set, and the test set. The training set is used to train the model, the validation set helps in tuning hyperparameters and assessing model performance, and the test set is used for final evaluation. A ML model architecture/algorithm is then chosen that is adapted for the problem being solved with machine learning. The problem can be classification, regression, clustering, or any other type of problem. The chosen model can be a decision tree, random forest, support vector machine, neural network, or any other model depending on the nature of the data and problem. The training set is then used to train the chosen model and the validation set is used to evaluate the model's performance. Once the model's performance is satisfactory, the model is evaluated using the test set. This provides an unbiased estimate of the model's performance and its ability to generalize to new data. Finally, the model can be deployed, and its performance can be monitored over time and adjustments or re-training made as needed. Machine learning metrics can quantify the performance of a machine learning model once it is already trained and can be used to select one or more potential trained models to use. The choice of metrics depends on the type of problem that is being solved (classification, regression, etc.) and include Accuracy, Precision, Recall, F1 Score, Area Under the ROC Curve (“AUC-ROC”), etc. SUMMARY Embodiments predict a target variable for accounts receivable using a machine learning model. Embodiments receive historical data corresponding to a plurality of transactions corresponding to a plurality of customers, the historical data comprising, for each of the transactions, the target variable. For a first customer, embodiments receive a plurality of trained ML models corresponding to the target variable, the plurality of trained ML models trained using the historical data and comprising a first trained model having no grace period for the target variable and two or more grace period trained models, each grace period trained model having different grace periods for the target variable. Embodiments determine a Matthews' Correlation Coefficient (“MCC”) for the first trained model. When the MCC for the first trained model is low, embodiments determine the MCC for each of the grace period trained models, and when one or more MCCs for each of the grace period trained models is higher than the MCC for the first trained model, embodiments select the corresponding grace period trained model having a highest MCC. When the first trained model or the selected grace period trained model has a high MCC, embodiments deploy the first trained model or the selected grace period trained model to predict the target variable. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various systems, methods, and other embodiments of the disclosure. It will be appreciated that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the figures represent one embodiment of the boundaries. In some embodiments one element may be designed as multiple elements or that multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component and vice versa. Furthermore, elements may not be drawn to scale. FIG. 1 illustrates an example of a system that includes a machine learning (“ML”) accounts receivable (“AR”) prediction model system in accordance to embodiments. FIG. 2 is a block diagram of the ML AR prediction model system of FIG. 1 in the form of a computer server/system in accordance to an embodiment of the present invention. FIG. 3 is a block diagram of a prediction system according to one embodiment. FIG. 4 is a block/flow diagram of a prediction system according to one embodiment for predicting AR related delays and highlighting the riskiest invoices and customers. FIG. 5 is a flow diagram of the ML AR prediction model module of FIG. 2 when predicting payment delays of AR payments in accordance to embodiments. FIG. 6 is a flow diagram of the ML AR prediction model m