KR-102963530-B1 - MACHINE LEARNING IN AUGMENTED REALITY CONTENT ITEMS

Abstract

The systems and methods of this specification describe receiving an image through an image capture device, using a machine learning model, generating an image augmentation decision, accessing an augmented reality content item, associating the generated image augmentation decision with the augmented reality content item, modifying the received image using the augmented reality content item and the associated image augmentation decision, and causing the presentation of the modified image on a graphical user interface of a computing device.

Inventors

• 라이클리우크, 올하
• 솔리친, 조나단
• 스톨리아, 알렉세이

Assignees

• 스냅 인코포레이티드

Dates

Publication Date

20260513

Application Date

20210607

Priority Date

20200619

Claims (20)

1. As a method, A step of receiving an image through an image capture device using one or more processors; A step of generating an image enhancement decision using a first machine learning model including a first model type—the image enhancement decision is based on the received image and non-image data received from one or more sensors of the image capture device—; A step of generating a supplementary image augmentation decision using a second machine learning model including a second model type—the supplementary image augmentation decision is based on the image augmentation decision, and the second model type is different from the first model type—; A step of accessing an augmented reality experience based on the above image augmentation decision and the above supplementary image augmentation decision; and A step of modifying the image using the augmented reality experience, the image augmentation decision, and the supplementary image augmentation decision. A method including
2. A method according to claim 1, wherein the non-image data comprises one or more of position data and audio data.
3. A method according to claim 1, further comprising the step of providing the generated image augmentation decision as an input to the augmented reality experience.
4. A method according to claim 1, wherein one or both of the first machine learning model and the second machine learning model are accessed from a resource library.
5. In paragraph 1, the first model type and the second model type are, A classification model configured to provide the probability that an input exists within a predefined category; A segmentation model configured to filter a portion of the image based on predefined criteria; and A feature (saliency) model configured to predict points of interest within the above image A method comprising one or more of the following.
6. A method according to claim 1, wherein the first model type is a segmentation model configured to filter a portion of the image based on a predefined criterion, and the second model type is one of a classification model configured to provide a probability that an input exists within a predefined category and a feature model configured to predict points of interest within the image.
7. A method according to claim 1, wherein the first model type is a segmentation model configured to filter a portion of the image based on predefined criteria, and the second machine learning model is configured to determine one or both of pattern and texture data of the portion of the image filtered by the first machine learning model.
8. In claim 1, the method wherein the augmented reality experience is automatically accessed without user intervention.
9. A method according to claim 1, wherein the augmented reality experience is an augmented reality content item configured to modify the image content of the received image.
10. As a system, One or more processors; and It includes memory storing instructions, and when the instructions are executed by the one or more processors, the one or more processors, The operation of receiving an image through an image capture device using one or more processors; An operation to generate an image enhancement decision using a first machine learning model including a first model type—the image enhancement decision is based on the received image and non-image data received from one or more sensors of the image capture device—; An operation to generate a supplementary image augmentation decision using a second machine learning model including a second model type - said supplementary image augmentation decision is based on said image augmentation decision, and said second model type is different from said first model type -; An action of accessing an augmented reality experience based on the above image augmentation decision and the above supplementary image augmentation decision; and A system that causes to perform operations including the augmented reality experience, the image augmentation decision, and the operation of modifying the image using the supplementary image augmentation decision.
11. A system according to claim 10, further comprising the operation of providing the generated image augmentation decision as an input to the augmented reality experience.
12. In paragraph 10, a system in which one or both of the first machine learning model and the second machine learning model are accessed from a resource library.
13. In Paragraph 10, The above first model type and the above second model type are, A classification model configured to provide the probability that an input exists within a predefined category; A segmentation model configured to filter a portion of the image based on predefined criteria; and A feature (saliency) model configured to predict points of interest within the above image A system including one or more of the following.
14. A system according to claim 10, wherein the first model type is a segmentation model configured to filter a portion of the image based on predefined criteria, and the second model type is one of a classification model configured to provide a probability that an input exists within a predefined category and a feature model configured to predict points of interest within the image.
15. A system according to claim 10, wherein the first model type is a segmentation model configured to filter a portion of the image based on predefined criteria, and the second machine learning model is configured to determine one or both of the pattern and texture data of the portion of the image filtered by the first machine learning model.
16. A non-transient computer-readable storage medium storing instructions, wherein the instructions, when executed by one or more processors of a machine, cause the machine to perform operations, and said operations, The operation of receiving an image through an image capture device using one or more processors; An operation to generate an image enhancement decision using a first machine learning model including a first model type—the image enhancement decision is based on the received image and non-image data received from one or more sensors of the image capture device—; An operation to generate a supplementary image augmentation decision using a second machine learning model including a second model type - said supplementary image augmentation decision is based on said image augmentation decision, and said second model type is different from said first model type -; An action of accessing an augmented reality experience based on the above image augmentation decision and the above supplementary image augmentation decision; and A non-transient computer-readable storage medium comprising the augmented reality experience, the image augmentation decision, and the operation of modifying the image using the supplementary image augmentation decision.
17. A non-transient computer-readable storage medium according to claim 16, further comprising the operation of providing the generated image augmentation decision as an input to the augmented reality experience.
18. In paragraph 16, a non-transient computer-readable storage medium in which one or both of the first machine learning model and the second machine learning model are accessed from a resource library.
19. In Clause 16, the above-mentioned first model type and the above-mentioned second model type are, A classification model configured to provide the probability that an input exists within a predefined category; A segmentation model configured to filter a portion of the image based on predefined criteria; and A feature (saliency) model configured to predict points of interest within the above image A non-transient computer-readable storage medium comprising one or more of the following.
20. A non-transient computer-readable storage medium, wherein the first model type is a segmentation model configured to filter a portion of the image based on predefined criteria, and the second model type is one of a classification model configured to provide a probability that an input exists within a predefined category and a feature model configured to predict points of interest within the image.

Description

Machine Learning in Augmented Reality Content Items This application claims the benefit of priority based on U.S. Patent Application No. 63/037,518 filed June 10, 2020 and U.S. Patent Application No. 16/946,413 filed June 19, 2020, each of which is incorporated herein by reference in its entirety. The embodiments of the present disclosure generally relate to machine learning. More specifically, without limitation, the present disclosure discloses systems and methods for using machine learning to provide decisions in augmented reality content items. Image augmentations, such as filters, allow users to curate creative and expressive content. Such augmentations provide users with generative tools to enhance the aesthetics of their content. Image augmentation capabilities are available through various social media tools. In many cases, users can provide user input to generate image augmentations. In drawings that are not necessarily drawn in fixed proportions, similar numerals may describe similar components in different drawings. To easily identify a discussion of any particular element or act, the top digit or numbers in the reference numbers refer to the drawing number where the element is first introduced. Some embodiments are illustrated in the drawings of the following accompanying drawings as examples rather than limitations: FIG. 1 is a schematic representation of a networked environment in which the present disclosure can be implemented, according to some examples. Figure 2 is a schematic representation of a messaging system having both client-side and server-side functions according to some examples. Figure 3 is a schematic representation of a data structure as maintained in a database, according to some examples. Figure 4 is a schematic representation of a message according to some examples. Figure 5 is a flowchart of an access restriction process according to some examples. FIG. 6 is a schematic representation of a machine learning model according to some exemplary embodiments. FIG. 7 is a flowchart of an exemplary method for generating customized image augmentations according to some exemplary embodiments. FIG. 8 is a schematic example of a modified image generated as a result of a custom augmentation system according to some exemplary embodiments. FIG. 9 is a schematic example of a modified image generated as a result of a custom augmentation system according to some exemplary embodiments. FIG. 10 is a schematic representation of a machine having the form of a computer system in which a set of instructions can be executed therein to cause the machine to perform any one or more methodologies discussed in this specification, according to some examples. FIG. 11 is a block diagram illustrating a software architecture in which examples can be implemented. The disclosed examples relate to a custom augmentation system that generates customized image augmentations using machine learning. The custom augmentation system may be used within augmented reality content items (e.g., augmented reality experiences, filters) to provide unique inputs to the augmented reality content items. Using this technique, the custom augmentation system may provide customized inputs to the augmented reality content items without receiving user input. Accordingly, the custom augmentation system uses machine learning models as logic providers for the augmented reality content items. In some examples, the custom augmentation system receives images via an image capture device. The custom augmentation system uses machine learning models to generate image augmentation decisions. In some examples, machine learning models are accessed from a resource library. The machine learning models can be segmentation models, classification models, object detection models, or saliency models. A segmentation model is a type of machine learning model that filters parts of an image based on specific criteria. A classification model is a type of model that provides the probability that input data exists within a specific category. An object detection model provides "bounding boxes" indicating where objects are located within the camera. A saliency model predicts points of interest within the image. The custom augmentation system generates image augmentation decisions based on the received image. The generated image augmentation decisions can be used to create more realistic image augmentations. In some examples, the generated image augmentation decisions are used to provide more contextual information. The custom augmentation system accesses an augmented reality content item. The augmented reality content item is configured to modify the image content of a received image. The custom augmentation system associates a generated image augmentation decision with the augmented reality content. For example, the custom augmentation system provides the generated image augmentation decision as input to the augmented reality content item. The custom augmentation