RU-2024133165-A - METHOD AND SYSTEM FOR AUTOMATIC CONTROL OF MACHINE LEARNING MODELS

Inventors

• Белозеров Максим Николаевич
• Смирнов Александр Николаевич

Assignees

• ПУБЛИЧНОЕ АКЦИОНЕРНОЕ ОБЩЕСТВО "СБЕРБАНК РОССИИ" (ПАО СБЕРБАНК)

Dates

Publication Date

20260505

Application Date

20241105

Claims (20)

1. 1. A method for automatically controlling machine learning models, performed by at least one computing device, comprising the steps of
2. receive a request from the user's device to provide a forecasting model to perform the forecasting task;
3. determining a list of models that can be used to perform the forecasting task, wherein the list of models contains at least: an identifier (id) of a first model and a first list of required parameters for forecasting; an id of a second model and a second list of required parameters for forecasting, wherein the second list of parameters contains at least one parameter different from the parameters from said first list of parameters;
4. determine the presence of marked data for the first parameter list and the second parameter list;
5. determine the methodology for training or retraining models from the list of models;
6. carry out training or retraining of the first model on the first list of parameters and the corresponding labeled data, and the second model on the second list of parameters and the corresponding labeled data;
7. determine the quality indicators of models from the list of models, taking into account the corresponding lists of parameters required by the models for forecasting;
8. determine the model with the highest quality score for the release of the trained model into production operation in the runtime environment.
9. 2. The method according to paragraph 1, characterized in that additional steps are performed in which it is determined that for at least one model from the list of models there is no labeled data, while training or further training of such a model is not carried out.
10. 3. The method according to paragraph 1, characterized in that additional steps are performed in which
11. analyze the quality indicators of models stored in the database;
12. determine the presence of a third model, the quality indicators of which are higher than the indicators of the first and second models, for the operation of which a third list of parameters is required;
13. determine the absence of at least one parameter from the third list of parameters in the available databases;
14. request from the user's device at least one parameter that is not present in the available databases, and the model ID is included in the list of models only if information about the mentioned parameter that is not present in the available databases has been provided.
15. 4. The method according to paragraph 1, characterized in that the determination of the quality indicators of the models is carried out taking into account the validation methodology determined for each model.
16. 5. The method according to paragraph 1, characterized in that the list of models that can be used to perform the forecasting task is determined based on the task ID.
17. 6. The method according to paragraph 1, characterized in that the following steps are additionally performed:
18. extract information about the required parameters for forecasting for each model;
19. The presence of the required parameters in the available databases is checked, and the model ID is included in the list of models if all the required parameters are contained in the available databases.
20. 7. The method according to paragraph 1, characterized in that the stage of determining the quality indicators of the models contains stages in which: