US-20260127632-A1 - Determining Winning Arms of A/B Electronic Communication Testing Using Resampling-Based Bayesian Nonparametrics

US20260127632A1US 20260127632 A1US20260127632 A1US 20260127632A1US-20260127632-A1

Abstract

Apparatuses, methods, and systems for determining winning arms of electronic testing. One method includes obtaining historical data values related to the A/B test of a user, storing the historical data values, determining a historical weight for the historical data values, receiving new data values from the plurality of computing devices collected based on recipient actions during execution of the A/B, constructing a Dirichlet distribution, inferring corresponding central tendencies of samplings of a metric distribution, wherein each central tendency of the corresponding central tendencies is determined by sampling the Dirichlet distribution, constructing an overall utility distribution for each arms of the A/B test by combining the central tendency of each sampling of the metric distribution with a corresponding sampling of a conversion probability distribution, determining a winning arm of the A/B testing by comparing the overall utility distribution of each arm with each other arm of the A/B test.

Inventors

Ian Delbridge

Assignees

KLAVIYO, INC.

Dates

Publication Date: 20260507
Application Date: 20251219

Claims (18)

1 . A system configured to select a winner of an A/B test, comprising: a plurality of computing devices; a user server; a management server, the management server connected to the user server and the plurality of computing devices through a network, the server configured to: obtain historical data values related to the A/B test of a user of the user server; store the historical data values; determine a historical weight for the historical data values; receive new data values from the plurality of computing devices collected based on recipient actions during execution of the A/B test and an associated metric; construct a Dirichlet distribution having one dimension for each of the historical data values and each of the received new data values, wherein distribution parameters of the Dirichlet distribution corresponding to the historical data values are the historical weight, and distribution parameters of the Dirichlet distribution corresponding to the received new data values are a new weight, wherein the new weight is at least as large as the historical weight; infer corresponding central tendencies of samplings of a metric distribution, wherein each central tendency of the corresponding central tendencies is determined by sampling the Dirichlet distribution; construct an overall utility distribution for each arms of the A/B test by combining the central tendency of each sampling of the metric distribution with a corresponding sampling of a conversion probability distribution; determine a winning arm of the A/B testing by comparing the overall utility distribution of each arm with each other arm of the A/B test; and electronically communicate with recipients of the computing devices using a template based on the winning arm of the A/B testing.
2 . A method for selecting a winner of an A/B test, comprising: obtaining historical data values related to the A/B test of a user of the user server; storing the historical data values; determining a historical weight for the historical data values; receiving new data values from the plurality of computing devices collected based on recipient actions during execution of the A/B test and an associated metric; constructing a Dirichlet distribution having one dimension for each of the historical data values and each of the received new data values, wherein distribution parameters of the Dirichlet distribution corresponding to the historical data values are the historical weight, and distribution parameters of the Dirichlet distribution corresponding to the received new data values are a new weight, wherein the new weight is at least as large as the historical weight; inferring corresponding central tendencies of samplings of a metric distribution, wherein each central tendency of the corresponding central tendencies is determined by sampling the Dirichlet distribution; constructing an overall utility distribution for each arms of the A/B test by combining the central tendency of each sampling of the metric distribution with a corresponding sampling of a conversion probability distribution; determining a winning arm of the A/B testing by comparing the overall utility distribution of each arm with each other arm of the A/B test; and electronically communicating with recipients of the computing devices using a template based on the winning arm of the A/B testing.
3 . The method of claim 2 , wherein each central tendency is determined for each sample of sampling of the Dirichlet distribution.
4 . The method of claim 2 , wherein the winning arm is an arm that most frequently has a highest sampled utility when sampling the overall utility distribution of the arm.
5 . The method of claim 2 , wherein the A/B test comprises testing of at least one of websites, forms, templates of emails, or templates of mobile messages.
6 . The method of claim 2 , wherein each arm of the A/B testing includes a template of at least one of a form, an email, or a mobile message, including an A template and a B template, wherein each template includes a set of data objects that combine to represent a structure of the form, the email, or the mobile message, wherein the A template and the B template of the form, the email, or the mobile message each have a different content, behavior, or send time.
7 . The method of claim 6 , wherein the different behavior includes the form, the email, or the mobile message popping up being loaded or sliding out from a side of a display of a recipient after being loaded.
8 . The method of claim 6 , further comprising selecting the winning arm as a user default, wherein the selected user default is used for electronic communications with recipients.
9 . The method of claim 2 , wherein the associated metric includes at least one of revenue per message, a purchase value, a quantity of items purchased, a number of times software users open an application, an amount of time users spends in an application.
10 . The method of claim 2 , further comprising selecting weighting of distribution of recipients receiving each of the arms based on the overall utility distribution for each of the arms of the A/B test.
11 . The method of claim 10 , wherein the weighting of distribution of each arm is based on how frequently the arm was a highest sampled utility when sampling the overall utility distribution of the arm.
12 . The method of claim 2 , further comprising ending the A/B test based on the overall utility distribution for each of the arms of the A/B test.
13 . The method of claim 2 , wherein the A/B testing is ended when the overall utility distribution of one arm is a threshold better than each of other arms.
14 . The method of claim 2 , wherein inferring corresponding central tendencies of samplings of a metric distribution comprises: sampling the Dirichlet distribution to create values for each data value; and computing a sample mean for each data value comprising summing a data value multiplied by a corresponding created value of the data value.
15 . The method of claim 2 , wherein receiving the new data collected based on the execution of the A/B test and the associated metric comprises sensing actions of recipients of electronic communications of the A/B test.
16 . The method of claim 15 , wherein sensing actions of the recipients includes sensing physical motion of the recipients including sensing keyboards actions of the recipients, sensing physical motion of the recipients, and tracking locations of the recipients.
17 . The system of claim 1 , wherein inferring corresponding central tendencies of samplings of a metric distribution comprises: sampling the Dirichlet distribution to create values for each data value; and computing a sample mean for each data value comprising summing a data value multiplied by a corresponding created value of the data value.
18 . The system of claim 1 , wherein receiving the new data collected based on the execution of the A/B test and the associated metric comprises sensing actions of recipients of electronic communications of the A/B test.

Description

RELATED PATENT APPLICATIONS This patent application is a continuation of U.S. patent application Ser. No. 18/588,109, filed Feb. 27, 2024, which is a continuation-in-part (CIP) of U.S. patent application Ser. No. 18/539,444, filed Dec. 14, 2023 and granted as U.S. patent Ser. No. 12/387,236 on Jul. 30, 2025, which is a continuation of U.S. patent application Ser. No. 18/201,644, filed May 24, 2023 and granted as U.S. patent Ser. No. 11/887,149, on Jan. 30, 2024, which are all herein incorporated by reference. FIELD OF THE DESCRIBED EMBODIMENTS The described embodiments relate generally to intelligent electronic communication management. More particularly, the described embodiments relate to systems, methods, and apparatuses for determining winning arms of electronic testing using resampling-based Bayesian nonparametrics. BACKGROUND Typically, A/B testing systems of messages limit the number of metrics that can be used to determine which variation of the A/B test wins. The variations are limited to a subset of metrics that have been characterized manually, or users of the system are solicited for information, such as, an expected conversion rate, minimum detectable sample size, and/or desired power. When limiting the variation to a subset, the users may not be able to optimize messages for the metrics that matter the most. When soliciting information from the users, the users carry a large cognitive load and must answer questions they may not be able to answer. Moreover, non-binary metrics like revenue—in contrast with binary metrics like conversion rates—are especially difficult to model probabilistically, even when only modeling a single metric, much less specifying a model flexible enough to handle any metric. It is desirable to have methods, apparatuses, and systems determining winning arms of electronic testing using resampling-based Bayesian nonparametrics. SUMMARY An embodiment includes a computer-implemented method for selecting a winner of an A/B test. The method includes obtaining historical data values related to the A/B test of a user of the user server, storing the historical data values, determining a historical weight for the historical data values, receiving new data values from the plurality of computing devices collected based on recipient actions during execution of the A/B test and an associated metric, constructing a Dirichlet distribution having one dimension for each of the historical data values and each of the received new data values, wherein distribution parameters of the Dirichlet distribution corresponding to the historical data values are the historical weight, and distribution parameters of the Dirichlet distribution corresponding to the received new data values are a new weight, wherein the new weight is at least as large as the historical weight, inferring corresponding central tendencies of samplings of a metric distribution, wherein each central tendency of the corresponding central tendencies is determined by sampling the Dirichlet distribution, constructing an overall utility distribution for each arms of the A/B test by combining the central tendency of each sampling of the metric distribution with a corresponding sampling of a conversion probability distribution, determining a winning arm of the A/B testing by comparing the overall utility distribution of each arm with each other arm of the A/B test; and electronically communicating with recipients of the computing devices using a template based on the winning arm of the A/B testing. Another embodiment includes a system configured to select a winner of an A/B test. The system includes a plurality of computing devices, a user server, and aa management server, wherein the management server connected to the user server and the plurality of computing devices through a network. The server is configured to obtain historical data values related to the A/B test of a user of the user server; store the historical data values, determine a historical weight for the historical data values, receive new data values from the plurality of computing devices collected based on recipient actions during execution of the A/B test and an associated metric, construct a Dirichlet distribution having one dimension for each of the historical data values and each of the received new data values, wherein distribution parameters of the Dirichlet distribution corresponding to the historical data values are the historical weight, and distribution parameters of the Dirichlet distribution corresponding to the received new data values are a new weight, wherein the new weight is at least as large as the historical weight, infer corresponding central tendencies of samplings of a metric distribution, wherein each central tendency of the corresponding central tendencies is determined by sampling the Dirichlet distribution, construct an overall utility distribution for each arms of the A/B test by combining the central tendency of each sampling of the metric