US-12626280-B2 - Methods, systems, and devices for counterfactual-based incrementality measurement in digital ad-bidding platform

US12626280B2US 12626280 B2US12626280 B2US 12626280B2US-12626280-B2

Abstract

A digital ad-buying platform uses counterfactual-based incrementality measurement by implementing randomization and/or a correction for auction win bias to avoid the need to identify counterfactual winner types in the control group. This approach can estimate impact at the individual consumer level. Confidence levels can be determined using Gibbs sampling in the context of causal analysis in the presence of non-compliance.

Inventors

Prasad Chalasani
Ari Buchalter
Ezra Winston
Jaynth Thiagarajan

Assignees

MediaMath Acquisition Corporation

Dates

Publication Date: 20260512
Application Date: 20221214

Claims (20)

1 . A method for implementation by one or more computing devices comprising: receiving, by a client device from an ad server, an online advertisement for insertion into an ad impression opportunity; and causing, by the client device, the online advertisement to be inserted into a webpage or video being displayed on a display device associated with the client device and corresponding to the ad impression opportunity; wherein the online advertisement is selected using a bidding method for adjusting future bid requests based on a causal ad impact of previous bid requests, the bidding method comprising: receiving, by a computer system of a demand side platform over a computer network, data encapsulating a first bid request from an ad exchange, wherein the first bid request is indicative of a first bidding opportunity at the ad exchange, the ad exchange being an online advertisement exchange in which digital advertisements are inserted into websites and/or online videos for consumption by a plurality of users; matching, by the computer system of the demand side platform, ad campaigns of advertisers to the first bid request to identify a subset list of advertisers eligible to submit a bid to the first bidding opportunity, the ad campaigns each comprising a plurality of digital advertisements to be inserted into websites and/or videos; executing, by the computer system of the demand side platform, a pre-bid randomization scheme prior to submitting bids, whereby each of the subset list of advertisers are categorized in a control group or a test group; logging, by the computer system of the demand side platform, a phantom control impression, the phantom control impression comprising first user identifiers associated with consumers, wherein the ad exchange does not transmit advertisements from ad campaigns associated with the list of advertisers categorized as the control group to the first user identifiers; submitting, by the computer system of the demand side platform, the bids associated with ad campaigns from the subset list of advertisers categorized as the test group; receiving, by the computer system of the demand side platform, results of the bid submissions, wherein the results indicate whether the bid was won or lost; logging, by the computer system of the demand side platform, a test-win impression, the test-win impression comprising second user identifiers associated with consumers, wherein the ad exchange transmits advertisements from ad campaigns associated with the list of advertisers categorized as the test group that won the bid to the third user identifiers; logging, by the computer system of the demand side platform, a test-lost impression, the test-lost impression comprising third user identifiers associated with consumers, wherein the ad exchange does not transmit advertisements from ad campaigns associated with the list of advertisers categorized as the test group that lost the bid to the second user identifiers; accessing, by the computer system of the demand side platform, cookie data associated with the first user identifiers of the phantom control impression, the second user identifiers of the test-win impression, and the third user identifiers of the test-lost impression, wherein the cookie data comprises a cookie ID that can be utilized to identify browser actions of the associated consumers; determining, by the computer system of the demand side platform, the causal ad impact based on the browser actions associated with the phantom control impression, the test-win impression, and the test-lost impression; and allowing advertisers to utilize the causal ad impact in a second bidding opportunity; wherein at least some of the bids are based on an efficacy of the online advertisement as generated by a machine learning model which is trained and implemented to calculate incrementality on an impression level.
2 . The method of claim 1 , wherein each of the plurality of users is categorized as the control group or the test group based on a fraction probability.
3 . The method of claim 1 , wherein the bidding method further comprises: identifying, by the computer system, one or more consumer responses based on the actions associated with the control group, the first sub-group, and the second sub-group, wherein determining the causal ad impact is further based at least in part on the one or more consumer responses.
4 . The method of claim 3 , wherein the one or more consumer responses comprises one or more of: a site visit, a registration, a subscription, an addition of items to a shopping cart, or a purchase.
5 . The method of claim 1 , wherein the computer system comprises a demand side platform.
6 . The method of claim 1 , wherein the bidding method further comprises: determining, by the computer system, a confidence factor for the causal ad impact, wherein the confidence factor comprises an interval and/or a value range associated with a probability percentile.
7 . The method of claim 6 , wherein the confidence factor is determined based at least in part on Markov-Chain Monte-Carlo sampling.
8 . The method of claim 7 , wherein the Markov-Chain Monte-Carlo sampling comprises a Gibbs sampling scheme.
9 . A method for implementation by one or more computing devices comprising: receiving, by a client device from an ad server, an online advertisement for insertion into an ad impression opportunity; and causing, by the client device, the online advertisement to be inserted into a webpage or video being displayed on a display device associated with the client device and corresponding to the ad impression opportunity; wherein the online advertisement is determined using a bidding method for determining and submitting one or more bids for an ad impression opportunity based on a causal ad impact of one or more previous bids, the bidding method comprising: receiving, by a computer system, a first bid request for a first ad impression opportunity; identifying, by the computer system, one or more advertisers to submit a bid in response to the first bid request to place one or more advertisements on user computing systems of a plurality of users; executing, by the computer system, a randomization scheme, whereby each of the plurality of users is categorized in one of a control group or a test group; logging, by the computer system, a control impression for the control group, wherein the one or more advertisements is not transmitted to user computing systems of the control group; submitting, by the computer system, one or more bids for placing the one or more advertisements on user computing systems of the test group; receiving, by the computer system, results of the one or more submitted bids, wherein the results indicate whether each of the one or more submitted bids was won or lost; logging, by the computer system, a test-win impression for a first sub-group of the test group, wherein at least one of the one or more advertisements is transmitted to user computing systems of the first sub-group; logging, by the computer system, a test-lost impression for a second sub-group, wherein the one or more advertisements is not transmitted to user computing systems of the second sub-group; accessing, by the computer system, data associated with the control group, the first sub-group, and the second sub-group; identifying, by the computer system, actions associated with the control group, the first sub-group, and the second sub-group by utilizing the accessed data; determining, by the computer system, the causal ad impact based at least in part on the identified actions associated with the control group, the first sub-group, and the second sub-group; analyzing, by the computer system using a machine learning model, the determined causal ad impact by determining the relative efficacy of the one or more submitted bids and determining similarities in characteristics between the one or more submitted bids having similar causal ad impact; providing, by the computer system, based on the analysis of the determined causal ad impact, one or more recommendations to the one or more advertisers related to submission of one or more bids in response to a second bid request for a second ad impression opportunity; adjusting, based on the determined causal ad impact and the provided one or more recommendations, the one or more bids in response to the second bid request for the second ad impression opportunity; and submitting, by the computer system, the adjusted one or more bids in response to the second bid request for the second ad impression opportunity; wherein at least some of the bids are based on an efficacy of the online advertisement as generated by a machine learning model which is trained and implemented to calculate incrementality on an impression level.
10 . The method of claim 9 , wherein each of the plurality of users is categorized as the control group or the test group based on a fraction probability.
11 . The computer-implemented method of claim 9 , wherein the bidding method further comprises: identifying, by the computer system, one or more consumer responses based on the actions associated with the control group, the first sub-group, and the second sub-group, wherein determining the causal ad impact is further based at least in part on the one or more consumer responses.
12 . The method of claim 11 , wherein the one or more consumer responses comprises one or more of: a site visit, a registration, a subscription, an addition of items to a shopping cart, or a purchase.
13 . The method of claim 9 , wherein the computer system comprises a demand side platform.
14 . The method of claim 9 , wherein the bidding method further comprises: determining, by the computer system, a confidence factor for the causal ad impact, wherein the confidence factor comprises an interval and/or a value range associated with a probability percentile.
15 . The method of claim 14 , wherein the confidence factor is determined based at least in part on Markov-Chain Monte-Carlo sampling.
16 . The computer-implemented method of claim 15 , wherein the Markov-Chain Monte-Carlo sampling comprises a Gibbs sampling scheme.
17 . A system comprising: at least one processor; and memory storing instructions which, when executed by the at least one data processor, results in operations comprising: receiving an online advertisement for insertion into an ad impression opportunity; and causing the online advertisement to be inserted into a webpage or video being displayed on a display device and corresponding to the ad impression opportunity; wherein the online advertisement is selected using a bidding method for adjusting future bid requests based on a causal ad impact of previous bid requests, the bidding method comprising: receiving, by a computer system of a demand side platform over a computer network, data encapsulating a first bid request from an ad exchange, wherein the first bid request is indicative of a first bidding opportunity at the ad exchange, the ad exchange being an online advertisement exchange in which digital advertisements are inserted into websites and/or online videos for consumption by a plurality of users; matching, by the computer system of the demand side platform, ad campaigns of advertisers to the first bid request to identify a subset list of advertisers eligible to submit a bid to the first bidding opportunity, the ad campaigns each comprising a plurality of digital advertisements to be inserted into websites and/or videos; executing, by the computer system of the demand side platform, a pre-bid randomization scheme prior to submitting bids, whereby each of the subset list of advertisers are categorized in a control group or a test group; logging, by the computer system of the demand side platform, a phantom control impression, the phantom control impression comprising first user identifiers associated with consumers, wherein the ad exchange does not transmit advertisements from ad campaigns associated with the list of advertisers categorized as the control group to the first user identifiers; submitting, by the computer system of the demand side platform, the bids associated with ad campaigns from the subset list of advertisers categorized as the test group; receiving, by the computer system of the demand side platform, results of the bid submissions, wherein the results indicate whether the bid was won or lost; logging, by the computer system of the demand side platform, a test-win impression, the test-win impression comprising second user identifiers associated with consumers, wherein the ad exchange transmits advertisements from ad campaigns associated with the list of advertisers categorized as the test group that won the bid to the third user identifiers; logging, by the computer system of the demand side platform, a test-lost impression, the test-lost impression comprising third user identifiers associated with consumers, wherein the ad exchange does not transmit advertisements from ad campaigns associated with the list of advertisers categorized as the test group that lost the bid to the second user identifiers; accessing, by the computer system of the demand side platform, cookie data associated with the first user identifiers of the phantom control impression, the second user identifiers of the test-win impression, and the third user identifiers of the test-lost impression, wherein the cookie data comprises a cookie ID that can be utilized to identify browser actions of the associated consumers; determining, by the computer system of the demand side platform, the causal ad impact based on the browser actions associated with the phantom control impression, the test-win impression, and the test-lost impression; and allowing advertisers to utilize the causal ad impact in a second bidding opportunity; wherein at least some of the bids are based on an efficacy of the online advertisement as generated by a machine learning model which is trained and implemented to calculate incrementality on an impression level.
18 . The system of claim 17 , wherein each of the plurality of users is categorized as the control group or the test group based on a fraction probability.
19 . The system of claim 1 , wherein the bidding method further comprises: identifying, by the computer system, one or more consumer responses based on the actions associated with the control group, the first sub-group, and the second sub-group, wherein determining the causal ad impact is further based at least in part on the one or more consumer responses.
20 . The system of claim 3 , wherein the one or more consumer responses comprises one or more of: a site visit, a registration, a subscription, an addition of items to a shopping cart, or a purchase.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. patent application Ser. No. 17/448,974 filed on Sep. 27, 2021, which is a continuation of U.S. patent application Ser. No. 17/188,469 filed on Mar. 1, 2021, now U.S. Pat. No. 11,170,413. Issued Nov. 9, 2021, which is a continuation of U.S. patent application Ser. No. 16/425,309 filed on May 29, 2019, now U.S. Pat. No. 10,977,697, issued Apr. 13, 2021, which is a continuation of U.S. patent application Ser. No. 15/667,507, filed on Aug. 2, 2017, now U.S. Pat. No. 10,467,659, issued Nov. 5, 2019 which claims the benefit of U.S. Provisional Patent Application No. 62/370,614, filed on Aug. 3, 2016, and U.S. Provisional Patent Application No. 62/491,522, filed on Apr. 28, 2017. Each of the foregoing applications is hereby incorporated herein by reference in its entirety for all purposes. BACKGROUND Field The embodiments disclosed herein generally relate to systems and methods for providing and/or improving a digital ad-buying platform system, and more particularly for counterfactual-based incrementality measurement in the digital ad-buying platform system. SUMMARY Various embodiments described herein disclose a digital ad-buying platform that can use counterfactual-based incrementality measurement by implementing randomization and/or a correction for auction win bias to avoid the need to identify counterfactual winner types in the control group. This approach can estimate impact at the individual consumer level. Confidence levels can be determined using Gibbs sampling in the context of causal analysis in the presence of non-compliance. In some embodiments, a computer-implemented method for adjusting future bid requests based on a causal ad impact of previous bid requests comprises: receiving, by a computer system of a demand side platform, a first bid request from an ad exchange, wherein the first bid request is indicative of a first bidding opportunity at the ad exchange; matching, by the computer system of the demand side platform, ad campaigns of advertisers to the first bid request to identify a subset list of advertisers eligible to submit a bid to the first bidding opportunity; executing, by the computer system of the demand side platform, a pre-bid randomization scheme prior to submitting bids, whereby each of the subset list of advertisers are categorized in a control group or a test group; logging, by the computer system of the demand side platform, a phantom control impression, the phantom control impression comprising first user identifiers associated with consumers, wherein the ad exchange does not transmit advertisements from ad campaigns associated with the list of advertisers categorized as the control group to the first user identifiers; submitting, by the computer system of the demand side platform, the bids associated with ad campaigns from the subset list of advertisers categorized as the test group; receiving, by the computer system of the demand side platform, results of the bid submissions, wherein the results indicate whether the bid was won or lost; logging, by the computer system of the demand side platform, a test-win impression, the test-win impression comprising second user identifiers associated with consumers, wherein the ad exchange transmits advertisements from ad campaigns associated with the list of advertisers categorized as the test group that won the bid to the third user identifiers; logging, by the computer system of the demand side platform, a test-lost impression, the test-lost impression comprising third user identifiers associated with consumers, wherein the ad exchange does not transmit advertisements from ad campaigns associated with the list of advertisers categorized as the test group that lost the bid to the second user identifiers; accessing, by the computer system of the demand side platform, cookie data associated with the first user identifiers of the phantom control impression, the second user identifiers of the test-win impression, and the third user identifiers of the test-lost impression, wherein the cookie data comprises a cookie ID that can be utilized to identify browser actions of the associated consumers; determining, by the computer system of the demand side platform, the causal ad impact based on the browser actions associated with the phantom control impression, the test-win impression, and the test-lost impression; and allowing advertisers to utilize the causal ad impact in a second bidding opportunity, wherein the computer system comprises a computer processor and an electronic storage medium. In certain embodiments, each of the subset list of advertisers are categorized as the control group or the test group based on a fraction probability, and wherein the causal ad impact is based on the fraction probability. In certain embodiments, the computer-implemented method further comprises: identifying, by the computer system of the demand side platform, a consumer respon