US-12619998-B2 - Predicting marketing outcomes using contrastive learning

US12619998B2US 12619998 B2US12619998 B2US 12619998B2US-12619998-B2

Abstract

Techniques for predicting marketing outcomes using contrastive learning are disclosed, including: obtaining historical marketing messages; obtaining historical open rates associated respectively with the historical marketing messages; based on the historical marketing messages, generating latent space representations associated respectively with the historical marketing messages; based on the latent space representations and respective contents of the historical marketing messages, training a first machine learning model to map contents of marketing messages to corresponding latent space representations of the marketing messages; based at least on the latent space representations and the historical open rates, training a second machine learning model to map latent space representations of marketing messages to predicted open rates of the marketing messages.

Inventors

Karempudi V. Ramarao

Assignees

ORACLE INTERNATIONAL CORPORATION

Dates

Publication Date: 20260505
Application Date: 20220916

Claims (13)

1 . One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause performance of operations comprising: obtaining a plurality of historical marketing messages; obtaining a plurality of historical open rates associated respectively with the plurality of historical marketing messages; based on non-content properties of the plurality of historical marketing messages and not based on respective contents of the plurality of marketing messages, generating a plurality of latent space representations associated respectively with the plurality of historical marketing messages; wherein first message content of a first historical marketing message in the plurality of historical marketing messages is more semantically similar to second message content of a second historical marketing message in the plurality of historical marketing messages than to third message content of a third historical marketing message in the plurality of historical marketing messages; and wherein despite the first message content being more semantically similar to the second message content than to the third message content, a first latent space representation associated with the first historical marketing message is farther, in latent space, from a second latent space representation associated with the second historical marketing message than from a third latent space representation associated with the third historical marketing message; based on the plurality of latent space representations and respective contents of the plurality of historical marketing messages, training a first machine learning model to map contents of marketing messages to corresponding latent space representations of the marketing messages; based at least on the plurality of latent space representations and the plurality of historical open rates, training a second machine learning model to map latent space representations of marketing messages to predicted open rates of the marketing messages; applying the first machine learning model to a particular marketing message, to obtain a latent space representation of the particular marketing message; applying the second machine learning model to the latent space representation of the particular marketing message, to obtain a predicted open rate of the particular marketing message; based at least in part on the predicted open rate of the particular marketing message: generating, without user input, additional digital content comprising one or more of an image, a video, or audio content to include in the particular marketing message; and publishing, without user input, the particular marketing message along with the additional digital content over a network to one or more digital distribution targets.
2 . The one or more non-transitory machine-readable media of claim 1 , wherein training the first machine learning model to map contents of marketing messages to corresponding latent space representations of the marketing messages comprises: computing a square of a magnitude of a particular latent space representation in the plurality of latent space representations; over a plurality of epochs of machine learning: converting corresponding contents in the plurality of historical marketing messages to a vector having same dimensions as the particular latent space representation; computing a dot product of the vector with the particular latent space representation; wherein over the plurality of epochs, the machine learning targets convergence of the dot product with the square of the magnitude of the particular latent space representation.
3 . The one or more non-transitory machine-readable media of claim 1 , wherein training the first machine learning model and training the second machine learning model are performed at least partly in parallel.
4 . The one or more non-transitory machine-readable media of claim 1 , wherein generating the plurality of latent space representations comprises vectorizing the plurality of historical marketing messages.
5 . The one or more non-transitory machine-readable media of claim 1 , wherein the first machine learning model and the second machine learning model comprise respective layers in a single neural network.
6 . The one or more non-transitory machine-readable media of claim 1 , wherein the plurality of historical marketing messages comprises a plurality of historical marketing emails.
7 . One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause performance of operations comprising: obtaining a plurality of historical marketing messages; obtaining a plurality of historical open rates associated respectively with the plurality of historical marketing messages; based on non-content properties of the plurality of historical marketing messages and not based on respective contents of the plurality of marketing messages, generating a plurality of latent space representations associated respectively with the plurality of historical marketing messages; wherein first message content of a first historical marketing message in the plurality of historical marketing messages is more semantically similar to second message content of a second historical marketing message in the plurality of historical marketing messages than to third message content of a third historical marketing message in the plurality of historical marketing messages; and wherein despite the first message content being more semantically similar to the second message content than to the third message content, a first latent space representation associated with the first historical marketing message is farther, in latent space, from a second latent space representation associated with the second historical marketing message than from a third latent space representation associated with the third historical marketing message; based on the plurality of latent space representations and respective contents of the plurality of historical marketing messages, training a first machine learning model to map contents of marketing messages to corresponding latent space representations of the marketing messages; based at least on the plurality of latent space representations and the plurality of historical open rates, training a second machine learning model to map latent space representations of marketing messages to predicted open rates of the marketing messages; applying the first machine learning model to a particular marketing message, to obtain a latent space representation of the particular marketing message; applying the second machine learning model to the latent space representation of the particular marketing message, to obtain a predicted open rate of the particular marketing message; based at least in part on the predicted open rate of the particular marketing message: generating, without user input, additional digital content comprising one or more of an image, a video, or audio content to include in the particular marketing message; and publishing, without user input, the particular marketing message along with the additional digital content over a network to one or more digital distribution targets.
8 . The one or more non-transitory machine-readable media of claim 7 , wherein training the first machine learning model to map contents of marketing materials to corresponding latent space representations of the marketing materials comprises: computing a square of a magnitude of a particular latent space representation in the plurality of latent space representations; over a plurality of epochs of machine learning: converting corresponding contents in the plurality of historical marketing materials to a vector having same dimensions as the particular latent space representation; computing a dot product of the vector with the particular latent space representation; wherein over the plurality of epochs, the machine learning targets convergence of the dot product with the square of the magnitude of the particular latent space representation.
9 . The one or more non-transitory machine-readable media of claim 7 , wherein training the first machine learning model and training the second machine learning model are performed at least partly in parallel.
10 . The one or more non-transitory machine-readable media of claim 7 , wherein generating the plurality of latent space representations comprises vectorizing the plurality of historical marketing materials.
11 . The one or more non-transitory machine-readable media of claim 7 , wherein the first machine learning model and the second machine learning model comprise respective layers in a single neural network.
12 . The one or more non-transitory machine-readable media of claim 7 , wherein the predicted marketing outcomes of the marketing materials comprise predicted open rates of emails.
13 . The one or more non-transitory machine-readable media of claim 1 , wherein the plurality of latent space representations comprises a plurality of vectors of numbers corresponding to respective non-content properties of the plurality of historical marketing messages.

Description

TECHNICAL FIELD The present disclosure relates to predicting marketing outcomes. In particular, the present disclosure relates to predicting marketing outcomes using machine learning. BACKGROUND In general, it is very difficult to predict how successful a particular marketing campaign will be. For example, given a particular marketing email message, it is difficult to predict what the open rate for that message will be. One approach is to perform natural language processing (NLP) on the subject lines of email messages and attempt to correlate the semantics of each subject line to its open rate. However, the semantics of email subject lines rarely correlate to their respective open rates. Accordingly, typical approaches that rely on techniques such as regression modeling tend to be ineffective for predicting marketing outcomes. Approaches that rely on human judgment—for example, hiring a marketing expert to analyze prior campaigns and craft subject lines for new campaigns—tend to be ineffective for similar reasons; the semantics of email subject lines (i.e., those parts that are most readily discernible to human readers) rarely correlate to their respective open rates. NLP, regression modeling, and such tend to suffer similarly for other kinds of campaign predictions, such as predicting the click rate for a given call to action (e.g., a text and/or image-based call to action), rates of progression to different stages of a marketing funnel, etc. One approach to predicting marketing outcomes is described in U.S. patent application Ser. No. 17/100,525, titled “Techniques for Selecting Content to Include in User Communications,” filed Nov. 20, 2020, the entire contents of which are incorporated herein by reference. There, user-facing content (e.g., email subject lines) is used to generate a vocabulary of “token-sets,” where each token-set can be associated with a performance parameter that represents the impact that token-set has on the target outcome (e.g., open rate). By focusing on specific phrases, this approach overcomes some of the problems associated with semantic analysis, while still relying on analysis of user-facing content, i.e., specific phrases (e.g., single-word phrases, two-word phrases, three-word phrases, etc.) that the intended recipient is expected to see. The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. BRIEF DESCRIPTION OF THE DRAWINGS The embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings. References to “an” or “one” embodiment in this disclosure are not necessarily to the same embodiment and mean at least one. In the drawings: FIG. 1 shows a block diagram that illustrates an example of a system in accordance with one or more embodiments; FIG. 2 illustrates an example set of operations for predicting marketing outcomes using contrastive learning in accordance with one or more embodiments; FIG. 3 illustrates an example set of operations for generating latent space representations in accordance with one or more embodiments. FIG. 4 illustrates an example set of operations for training a contents-to-LSR model in accordance with one or more embodiments. FIGS. 5A-5B illustrate an example of predicting marketing outcomes using contrastive learning in accordance with one or more embodiments; and FIG. 6 shows a block diagram that illustrates a computer system in accordance with one or more embodiments. DETAILED DESCRIPTION In the following description, for the purposes of explanation and to provide a thorough understanding, numerous specific details are set forth. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some examples, well-known structures and devices are described with reference to a block diagram form, in order to avoid unnecessarily obscuring the present invention. The following table of contents is provided for reference purposes only and should not be construed as limiting the scope of one or more embodiments 1. GENERAL OVERVIEW2. EXAMPLE SYSTEM 2.1. SYSTEM COMPONENTS2.2. MARKETING PLATFORM2.3. DATA STORAGE2.4. USER INTERFACE2.5. TENANTS2.6. MACHINE LEARNING 3. PREDICTING MARKETING OUTCOMES USING CONTRASTIVE LEARNING 3.1. PROCESS OVERVIEW3.2. GENERATING LSRS OF HISTORICAL MARKETING MATERIALS3.3. TRAINING A CONTENTS-TO-LSR MODEL 4. EXAMPLE EMBODIMENT5. EXAMPLE SUBJECT LINES6. CONTENT DISTRIBUTION7. COMPUTER NETWORKS AND CLOUD NETWORKS8. MICROSERVICE APPLICATIONS 8.1. TRIGGERS8.2. ACTIONS 9. HARDWARE OVERVIEW10. MISCELLANEOUS; EXTENSIONS 1. General O