US-12626124-B2 - Dynamic user interface and machine learning tools for generating digital content and multivariate testing recommendations

US12626124B2US 12626124 B2US12626124 B2US 12626124B2US-12626124-B2

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods that utilize a dynamic user interface and machine learning tools to generate data-driven digital content and multivariate testing recommendations for distributing digital content across computer networks. In particular, in one or more embodiments, the disclosed systems utilize machine learning models to generate digital recommendations at multiple development stages of digital communications that are targeted on particular performance metrics. For example, the disclosed systems utilize historical information and recipient profile data to generate recommendations for digital communication templates, fragment variants of content fragments, and content variants of digital content items. Ultimately, the disclosed systems generate multivariate testing recommendations incorporating selected fragment variants to intelligently narrow multivariate testing candidates and generate more meaningful and statistically significant multivariate testing results.

Inventors

Eunyee Koh
Nedim Lipka
SHRIRAM VENKATESH SHET REVANKAR
Nikhil Belsare
Tak Yeon Lee
Andrew Thomson
Vasanthi Holtcamp
Ryan Rossi
Fan Du
Caroline Kim
Tong Yu
Shunan Guo

Assignees

ADOBE INC.

Dates

Publication Date: 20260512
Application Date: 20210722

Claims (20)

1 . A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to identify a set of digital communication variants for multivariate testing across a plurality of computing devices by: determining, utilizing a template recommendation model, digital communication templates based on predicted digital communication performance metrics for the digital communication templates; in response to selection, via a graphical user interface, of a digital communication template corresponding to a content fragment, generating, utilizing a fragment machine learning model, fragment variants for the content fragment based on predicted content fragment performance metrics for the fragment variants by: identifying previous digital communications comprising the selected digital communication template; determining, from viewing and interaction information associated with the previous digital communications, temporal dependencies and spatial dependencies between the content fragment and other content fragments in the selected digital communication template; and generating the fragment variants based on the temporal dependencies and the spatial dependencies; generating, utilizing a multivariate testing results prediction model, predicted multivariate performance metrics for combinations of content fragments including the fragment variants; generating multivariate testing recommendations based on the generated predicted multivariate performance metrics, each multivariate testing recommendation comprising a respective combination of content fragments and including a fragment variant; and modifying the graphical user interface to comprise two or more of the multivariate testing recommendations and respective performance metric indicator graphical elements, each respective performance metric indicator graphical element visually depicting one or more volumes that are filled in to a level reflective of and corresponding to one or more respective predicted multivariate performance metrics corresponding to the respective multivariate testing recommendation.
2 . The non-transitory computer-readable storage medium as recited in claim 1 , wherein determining, utilizing the template recommendation model, the digital communication templates comprises: identifying historical digital communications comprising digital communication contents and associated with historical performance data; determining, from the historical performance data, predicted digital communication performance metrics for digital communication templates corresponding to the historical digital communications; and providing digital communication templates utilizing the predicted digital communication performance metrics.
3 . The non-transitory computer-readable storage medium as recited in claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising: generating categories of the digital communication templates based on the digital communication contents of the associated historical digital communications; and providing the digital communication templates based on the generated categories.
4 . The non-transitory computer-readable storage medium as recited in claim 1 , wherein generating, utilizing the fragment machine learning model, the fragment variants for the content fragment comprises determining temporal dependencies by: generating a plurality of input vectors for the fragment machine learning model by encoding historical instantiation data reflecting user behaviors associated with the content fragment across multiple consecutive sessions; and utilizing a plurality of LSTM layers of the fragment machine learning model in connection with a model of historical viewing and interaction behavior relative to the content fragment to generate a predicted content item interaction performance metric for instantiations of the content fragment.
5 . The non-transitory computer-readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to, in response to selection of a digital content item within a selected fragment variant, utilize an image content model to generate digital content variants for the digital content item by: determining one or more descriptors for the digital content item; identifying digital content variants from the one or more descriptors; generating predicted digital content performance metrics for the digital content variants; and providing a subset of the digital content variants from the predicted digital content performance metrics.
6 . The non-transitory computer-readable storage medium as recited in claim 1 , wherein generating the multivariate testing recommendations comprises: generating predicted digital communication performance metrics for candidate digital communications comprising the fragment variants from historical performance data associated with historical digital communications; and generating the multivariate testing recommendations from a subset of the candidate digital communications based on the predicted digital communication performance metrics.
7 . The non-transitory computer-readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising: receiving a user interaction within the graphical user interface indicating an additional selection of a subset of the multivariate testing recommendations for live multivariate testing; and executing live multivariate testing on the additional selection of the subset of the multivariate testing recommendations by segmenting an audience of recipient client computing devices according to a number of the subset of the multivariate testing recommendations.
8 . A system comprising: one or more memory devices; and one or more processors configured to cause the system to identify a set of digital communication variants for multivariate testing across a plurality of computing devices by: determining, utilizing a template recommendation model, digital communication templates based on predicted digital communication performance metrics for the digital communication templates; in response to selection, via a graphical user interface, of a digital communication template corresponding to a content fragment, generating, utilizing a fragment machine learning model, fragment variants for the content fragment based on predicted content fragment performance metrics for the fragment variants by: identifying previous digital communications comprising the selected digital communication template; determining, from viewing and interaction information associated with the previous digital communications, temporal dependencies and spatial dependencies between the content fragment and other content fragments in the selected digital communication template; and generating the fragment variants based on the temporal dependencies and the spatial dependencies; generating, utilizing a multivariate testing results prediction model, predicted multivariate performance metrics for combinations of content fragments including the fragment variants; generating multivariate testing recommendations based on the generated predicted multivariate performance metrics, each multivariate testing recommendation comprising a respective combination of content fragments and including a fragment variant; and modifying the graphical user interface to comprise two or more of the multivariate testing recommendations and respective performance metric indicator graphical elements, each respective performance metric indicator graphical element visually depicting one or more volumes that are filled in to a level reflective of and corresponding to one or more respective predicted multivariate performance metrics corresponding to the respective multivariate testing recommendation.
9 . The system of claim 8 , wherein determining, utilizing the template recommendation model, the digital communication templates comprises: identifying historical digital communications comprising digital communication contents and associated with historical performance data; determining, from the historical performance data, predicted digital communication performance metrics for digital communication templates corresponding to the historical digital communications; and providing digital communication templates utilizing the predicted digital communication performance metrics.
10 . The system of claim 9 , wherein the one or more processors are further configured to cause the system to perform operations comprising: generating categories of the digital communication templates based on the digital communication contents of the associated historical digital communications; and providing the digital communication templates based on the generated categories.
11 . The system of claim 8 , wherein generating, utilizing the fragment machine learning model, the fragment variants for the content fragment comprises: generating an input vector for the fragment machine learning model by encoding historical instantiation data associated with the content fragment; and utilizing an LSTM layer of the fragment machine learning model in connection with a model of historical viewing and interaction behavior relative to the content fragment to generate a predicted content item interaction performance metric for an instantiation of the content fragment.
12 . The system of claim 8 , wherein the one or more processors are further configured to cause the system to, in response to selection of a digital content item within a selected fragment variant, utilize an image content model to generate digital content variants for the digital content item by: determining one or more descriptors for the digital content item; identifying digital content variants from the one or more descriptors; generating predicted digital content performance metrics for the digital content variants; and providing a subset of the digital content variants from the predicted digital content performance metrics.
13 . The system of claim 8 , wherein generating the multivariate testing recommendations comprises: generating predicted digital communication performance metrics for candidate digital communications comprising the fragment variants from historical performance data associated with historical digital communications; and generating the multivariate testing recommendations from a subset of the candidate digital communications based on the predicted digital communication performance metrics.
14 . The system of claim 8 , wherein the one or more processors are further configured to cause the system to perform operations comprising: receiving multivariate test performance metrics from a multivariate test of three or more of the multivariate testing recommendations; and generating a digital communication comprising a multivariate digital communication test recommendation from the multivariate test performance metrics.
15 . A computer-implemented method comprising identifying a set of digital communication variants for multivariate testing across a plurality of computing devices by: determining, utilizing a template recommendation model, digital communication templates based on predicted digital communication performance metrics for the digital communication templates; in response to selection, via a graphical user interface, of a digital communication template corresponding to a content fragment, generating, utilizing a fragment machine learning model, fragment variants for the content fragment based on predicted content fragment performance metrics for the fragment variants by: identifying previous digital communications comprising the selected digital communication template; determining, from viewing and interaction information associated with the previous digital communications, temporal dependencies and spatial dependencies between the content fragment and other content fragments in the selected digital communication template; and generating the fragment variants based on the temporal dependencies and the spatial dependencies; generating, utilizing a multivariate testing results prediction model, predicted multivariate performance metrics for combinations of content fragments including the fragment variants; generating multivariate testing recommendations based on the generated predicted multivariate performance metrics, each multivariate testing recommendation comprising a respective combination of content fragments and including a fragment variant; and modifying the graphical user interface to comprise two or more of the multivariate testing recommendations and respective performance metric indicator graphical elements, each respective performance metric indicator graphical element visually depicting one or more volumes that are filled in to a level reflective of and corresponding to one or more respective predicted multivariate performance metrics corresponding to the respective multivariate testing recommendation.
16 . The computer-implemented method of claim 15 , wherein determining, utilizing the template recommendation model, the digital communication templates comprises: identifying historical digital communications comprising digital communication contents and associated with historical performance data; determining, from the historical performance data, predicted digital communication performance metrics for digital communication templates corresponding to the historical digital communications; and providing digital communication templates utilizing the predicted digital communication performance metrics.
17 . The computer-implemented method of claim 15 , wherein generating, utilizing the fragment machine learning model, the fragment variants for the content fragment comprises: generating an input vector for the fragment machine learning model by encoding historical instantiation data associated with the content fragment; and utilizing an LSTM layer of the fragment machine learning model in connection with a model of historical viewing and interaction behavior relative to the content fragment to generate a predicted content item interaction performance metric for an instantiation of the content fragment.
18 . The computer-implemented method of claim 15 , further comprising, in response to selection of a digital content item within a selected fragment variant, utilizing an image content model to generate digital content variants for the digital content item by: determining one or more descriptors for the digital content item; identifying digital content variants from the one or more descriptors; generating predicted digital content performance metrics for the digital content variants; and providing a subset of the digital content variants from the predicted digital content performance metrics.
19 . The computer-implemented method of claim 15 , wherein generating the multivariate testing recommendations comprises: generating predicted digital communication performance metrics for candidate digital communications comprising the fragment variants from historical performance data associated with historical digital communications; and generating the multivariate testing recommendations from a subset of the candidate digital communications based on the predicted digital communication performance metrics.
20 . The computer-implemented method of claim 15 , further comprising: receiving multivariate test performance metrics from a multivariate test of three or more of the multivariate testing recommendations; and generating a digital communication comprising a multivariate digital communication test recommendation from the multivariate test performance metrics.

Description

BACKGROUND Recent years have seen significant improvements in hardware and software platforms for generating and distributing targeted digital communications across computer networks. For example, conventional systems can provide a variety of digital content and formatting selections that allow distribution devices to configure digital communications for distribution to various target client devices. To illustrate, conventional systems often provide an overwhelming variety of digital images and other media, fonts, and formatting options that a distribution device can configure into a digital communication for distribution to client devices. Once digital communications are configured utilizing selections from the available digital content, conventional systems also provide methods for testing the effectiveness of those digital communications. For example, conventional systems can monitor interactions from client devices to determine performance data associated with digital communications and provide analytical insights regarding the effectiveness of the digital communications. Although conventional systems can analyze the performance of versions digital communications, such systems have a number of problems in relation to accuracy, efficiency, and flexibility of operation. For instance, conventional systems often generate inaccurate or incomplete testing in connection with distributed digital communications. Specifically, when a digital communication campaign includes many potential versions of a digital communication, conventional systems test the potential versions of the digital communication. For instance, conventional systems generally split a recipient audience across potential digital communication versions, and then attempt to collect meaningful analytical data for each digital communication version. This is problematic, however, when the recipient audience segments are small due to a high number of potential digital communication versions—such as when there are multiple variable changes among each of the potential digital communication version. The ensuing performance test results provided by conventional systems are often inaccurate because they are limited to the actions of insignificant audience segments associated with each of the high number of digital communication versions. Conventional systems are also often rigid and inflexible. For example, some conventional systems utilize a bandit algorithm to select a combination of elements for a digital communication over time. This approach, however, generally focuses on a rigid reward associated with a particular content item. By focusing primarily on this reward, conventional systems fail to consider a variety of additional features or factors, such as spatial and temporal indicators, that significantly impact performance of distributed digital content. Additionally, conventional systems are inefficient. For example, as just discussed, conventional systems conduct digital performance analyses regarding a wide array of potential digital communication versions. This approach results in a significant expenditure of computing resources. For instance, some conventional systems require a significant number of user interface interactions to train models and/or select digital content. Moreover, conventional systems further waste vast amounts of computing resources (e.g., display resources, memory resources, processing resources, network resources) performing performance testing of high numbers of digital communication versions. These along with additional problems and issues exist with regard to conventional systems. BRIEF SUMMARY One or more embodiments described herein provide benefits and/or solve one or more of the foregoing or other problems in the art with systems, methods, and non-transitory computer-readable media that utilize a dynamic user interface and machine learning tools to generate data-driven digital content and multivariate testing recommendations for distributing digital content across computer networks. For example, the disclosed systems intelligently analyze the performance of historical digital communications utilizing a variety of machine learning models to generate recommendations at multiple stages of digital communication development and multivariate testing. To illustrate, the disclosed systems generate data-driven performance metrics in connection with digital communication templates to recommend specific templates for selection of digital content. The disclosed systems further utilize an HTML content machine learning model to generate internal design fragment content and recommendations based on a combination of historical digital communication performance data and design relevance. This fragment machine learning model can utilize a unique architecture that includes an LSTM model and graphical model to flexibly consider spatial and temporal dependencies and further improve accuracy in selecting digital content for distribution