US-12619487-B2 - Classifier validation

Abstract

One or more computing devices, systems, and/or methods for classifier validation are provided. A set of in-sample examples are partitioned into a reduced in-sample set and a remaining in-sample set. The reduced in-sample set is processed using a set of classifiers. A subset of classifiers are identified as having error counts, over the reduced in-sample set, below a threshold number of errors. A training procedure is executed to select a classifier having a minimum error rate over the set of in-sample examples. If the classifier is within the subset of classifiers, then an out-of-sample error bound is determined for the classifier.

Inventors

• Eric Theodore Bax
• Natalie Bax

Assignees

• YAHOO ASSETS LLC

Dates

Publication Date

20260505

Application Date

20221011

Claims (20)

1. 1 . A method, comprising: executing, on a processor of a computing device, instructions that cause the computing device to perform operations, the operations comprising: identifying a subset of classifiers of a set of classifiers as having error counts below a threshold number of errors; executing a training procedure to select a classifier from the set of classifiers based upon the classifier having an error rate less than an error rate threshold; upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon a second error rate of the classifier, wherein the determining an out-of-sample error bound increases a reliability of one or more classifiers used to perform one or more tasks; and automatically determining whether to deploy the classifier, for machine learning, based upon the out-of-sample error bound.
2. 2 . The method of claim 1 , wherein the determining the out-of-sample error bound comprises: determining the out-of-sample error bound based upon a count of remaining examples.
3. 3 . The method of claim 1 , wherein the determining the out-of-sample error bound comprises: determining the out-of-sample error bound based upon a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers.
4. 4 . The method of claim 1 , comprising: reporting a bound failure based upon the classifier not being in the subset of classifiers.
5. 5 . The method of claim 1 , wherein the identifying a subset of classifiers comprises: in response to the set of classifiers exceeding a threshold number of classifiers, sampling a portion of the set of classifiers as a set of sampled classifiers for identifying the subset of classifiers.
6. 6 . The method of claim 5 , wherein the identifying a subset of classifiers comprises: identifying a fraction of the set of sampled classifiers that have error counts below the threshold number of errors.
7. 7 . The method of claim 6 , wherein the identifying a subset of classifiers comprises: utilizing the fraction of the set of sampled classifiers to bound a rate at which classifiers in the set of classifiers have error counts below the threshold number of errors.
8. 8 . The method of claim 1 , wherein the identifying a subset of classifiers comprises: in response to the set of classifiers not exceeding a threshold number of classifiers, testing all classifiers within the set of classifiers for identifying classifiers having error counts below the threshold number of errors.
9. 9 . The method of claim 1 , comprising: in response to the out-of-sample error bound being within a tolerance threshold, utilizing the classifier to classify out-of-sample data.
10. 10 . The method of claim 1 , comprising: in response to the out-of-sample error bound exceeding a tolerance threshold, obtaining additional in-sample examples for training the set of classifiers.
11. 11 . A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising: identifying a subset of classifiers of a set of classifiers as having error counts below a threshold number of errors; executing a training procedure to select a classifier from the set of classifiers based upon the classifier having a minimum error rate; upon the classifier being in the subset of classifiers, determining an out-of-sample error bound for the classifier based upon an error rate of the classifier, a count of remaining examples, and a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers, wherein the determining an out-of-sample error bound increases a reliability of one or more classifiers used to perform one or more tasks; and automatically determining whether to deploy the classifier, for machine learning, based upon the out-of-sample error bound.
12. 12 . The non-transitory machine readable medium of claim 11 , wherein the operations comprise: in response to the out-of-sample error bound being within a tolerance threshold, utilizing the classifier to classify out-of-sample data.
13. 13 . A computing device comprising: a processor; and memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising: determining an upper bound on a probability that a classifier drawn according to a distribution has an error count below a threshold number of errors; performing a training procedure for weighted ensemble classifiers to select a posterior distribution over a set of classifiers; computing an out-of-sample error bound based upon a divergence between the distribution and the posterior distribution, wherein the computing an out-of-sample error bound increases a reliability of one or more classifiers used to perform one or more tasks; and automatically determining whether to deploy the classifier, for machine learning, based upon the out-of-sample error bound.
14. 14 . The computing device of claim 13 , wherein the probability is a shrinkage ratio, and wherein the computing an out-of-sample error bound comprises: adding a natural logarithm of the upper bound as the shrinkage ratio to the divergence.
15. 15 . The computing device of claim 13 , wherein the computing an out-of-sample error bound comprises: utilizing in-sample examples for computing the out-of-sample error bound.
16. 16 . The computing device of claim 13 , wherein the computing an out-of-sample error bound comprises: adding a fraction of the posterior distribution placed on classifiers that do not have error counts below the threshold number of errors to the out-of-sample error bound.
17. 17 . The computing device of claim 13 , wherein the out-of-sample error bound is a PAC-Bayes out-of-sample error bound.
18. 18 . The computing device of claim 13 , wherein the out-of-sample error bound is based upon a Kullback-Leibler divergence.
19. 19 . The computing device of claim 13 , wherein the operations comprise: upon a bound being within an acceptance threshold, utilizing the classifier on out-of-sample data.
20. 20 . The computing device of claim 13 , wherein the operations comprise: upon a bound not being within an acceptance threshold, obtaining additional in-sample data.

Description

RELATED APPLICATION This application claims priority to and is a continuation of U.S. application Ser. No. 17/034,427, filed on Sep. 28, 2020, entitled “CLASSIFIER VALIDATION”, which is incorporated by reference herein in its entirety. BACKGROUND Classifiers are utilized to perform various types of tasks, such as identifying objects within images (e.g., identifying a bone fracture within an input image), determining whether a user will have an interest in certain content (e.g., an interest in a topic of a video, image, article, etc.), classifying emails as spam or not spam, etc. There is a vast number of different types of classifiers that can be utilized to perform tasks, such as neural networks, decision trees, k-nearest neighbors, logical regression, Naive Bayes, etc. In order to select a classifier to utilize for a task, a set of classifiers are trained using in-sample examples. An in-sample example comprises an input-output pair with an input (e.g., an image) and a known/correct output (e.g., the image depicts a bone fracture). Various types of selection procedures can be utilized for selection a classifier. A classifier may be selected based upon the classifier having a relatively low error rate over the in-sample examples. Validation can be performed to determine whether the classifier should be used outside of training on out-of-sample data. Validation of the classifier can be based upon a machine learning error bound corresponding to an out-of-sample error rate. SUMMARY In accordance with the present disclosure, one or more computing devices and/or methods for classifier validation are provided. In some embodiments of classifier validation, a set of in-sample examples are partitioned into a reduced in-sample set of examples and a remaining in-sample set of examples. The reduced in-sample set is processed using a set of classifiers. A sub-set of classifiers of the set of classifiers are identified as having error counts below a threshold number of errors over the reduced in-sample set. If the number of classifiers within the set of classifiers is below a threshold, then the set of classifiers is utilized to identify the sub-set of classifiers. Otherwise, if the number of classifiers exceeds the threshold, then merely a sampling of the set of classifiers is utilized to identify the sub-set of classifiers. A training procedure is executed to select a classifier from the set of classifiers based upon the classifier having an error rate over the set of in-sample examples that is less than an error rate threshold (e.g., a minimum error rate compared to other classifiers). If the classifier is in the subset of classifiers, then an out-of-sample error bound is determined for the classifier. The out-of-sample error bound is determined based upon an error rate of the classifier over the remaining in-sample set, a count of remaining examples within the remaining in-sample set, and/or a ratio of a selected upper bound for a probability of bound failure and a number of classifiers in the subset of classifiers. In some embodiments of classifier validation, a set of in-sample examples are partitioned into a reduced in-sample set of examples and a remaining in-sample set of examples. A determination is made as to an upper bound on a probability that a classifier drawn according to a distribution has an error count over the reduced in-sample set that is below a threshold number of errors. A training procedure is performed for weighted ensemble classifiers to select a posterior distribution over the set of classifiers using the set of in-sample examples. An out-of-sample error bound is computed based upon a divergence between the distribution and the posterior distribution. The out-of-sample error bound may also be computed by adding a natural logarithm of the upper bound as a shrinkage ratio (e.g., the shrinkage ratio corresponding to the probability that a classifier drawn according to the distribution has the error count over the reduced in-sample that is below a threshold number of errors set) to the divergence. The out-of-sample error bound may also be computed utilizing the remaining in-sample set as in-sample examples for computing the out-of-sample error bound. The out-of-sample error bound may also be computed by adding a fraction of the posterior distribution placed on classifiers that do not have error counts below the threshold number of errors to the out-of-sample error bound. DESCRIPTION OF THE DRAWINGS While the techniques presented herein may be embodied in alternative forms, the particular embodiments illustrated in the drawings are only a few examples that are supplemental of the description provided herein. These embodiments are not to be interpreted in a limiting manner, such as limiting the claims appended hereto. FIG. 1 is an illustration of a scenario involving various examples of networks that may connect servers and clients. FIG. 2 is an illustration of a scenario involving an example config