Search

KR-20260064428-A - VIRTUAL MAKEUP SOLUTION PROVIDING SYSTEM USING GENERATIVE ARTIFICIAL INTELLIGENCE

KR20260064428AKR 20260064428 AKR20260064428 AKR 20260064428AKR-20260064428-A

Abstract

A system for providing a virtual makeup solution using generative AI is provided, comprising: a user terminal that captures a face and outputs a makeup image with virtual makeup applied to the face; a receiver that receives the face image from the user terminal; a feature extraction unit that analyzes and classifies the face image using a pre-established feature extraction model and outputs face feature information; a color extraction unit that inputs the face feature information into a pre-established LLM (Large Language Model)-based generative AI (Generative Artificial Intelligence) to extract a makeup color corresponding to the face feature information; and a virtual makeup unit that outputs a makeup image with virtual makeup applied using a diffusion-based generative AI that implants the extracted makeup color into the face image.

Inventors

  • 정성민
  • 고명진
  • 전희찬
  • 박건영
  • 양세린
  • 한인화
  • 예종철

Assignees

  • (주)아모레퍼시픽
  • 한국과학기술원

Dates

Publication Date
20260507
Application Date
20250103
Priority Date
20241031

Claims (10)

  1. A user terminal that photographs a face and outputs a makeup image with virtual makeup applied to the face; and A makeup service providing server comprising: a receiving unit that receives a face image from the above-mentioned user terminal; a feature extraction unit that analyzes and classifies the face image using a pre-established feature extraction model and outputs face feature information; a color extraction unit that inputs the face feature information into a pre-established LLM (Large Language Model)-based Generative Artificial Intelligence (GAI) to extract a makeup color corresponding to the face feature information; and a virtual makeup unit that outputs a virtually made-up makeup image using a diffusion-based Generative Artificial Intelligence that implants the extracted makeup color into the face image. A virtual makeup solution providing system using generative AI including
  2. In Article 1, The feature extraction model constructed above is, A virtual makeup solution providing system using generative AI that recognizes facial contours and facial components, selects facial feature points to analyze the length, proportion, and angle of each part, and uses BiseNet (Bilateral Segmentation Network for Real-time Semantic Segmentation) to extract facial skin tone, lip color, and eye color.
  3. In Article 2, The feature extraction model constructed above is, A system for providing a virtual makeup solution using generative AI, characterized by using ResNet (Residual Neural Network), a classification model that classifies faces based on pre-set classification items, based on the facial contours, facial components, facial feature points, lengths, ratios, and angles of each part, facial skin tone, lip color, and eye color.
  4. In Article 1, The above-mentioned LLM-based generative AI is, A virtual makeup solution providing system using a generative AI characterized by being trained using instruction tuning with a dataset of facial feature information and makeup colors, and then outputting keywords of the makeup colors when the facial feature information is input into the LLM-based generative AI.
  5. In Article 1, The above diffusion-based generative AI is, A virtual makeup solution providing system using generative AI characterized by being a Stable Diffusion, which is a Latent Diffusion Model.
  6. In Article 5, The above diffusion-based generative AI is, A virtual makeup solution providing system using generative AI, characterized by converting keywords of makeup colors output from the above LLM-based generative AI into prompts to generate the above makeup image.
  7. In Article 5, The above diffusion-based generative AI is, A system for providing a virtual makeup solution using generative AI, characterized by dividing the above-mentioned face image into pre-set makeup areas, generating a unit makeup image by converting a keyword of the above-mentioned makeup color into a prompt in the divided makeup area, and then generating the above-mentioned makeup image by inserting the unit makeup image back into the original face image to replace it.
  8. In Article 1, The above makeup service providing server is, A selection reflection unit that, after a makeup color corresponding to the above facial feature information is extracted, transmits a lip color among the makeup colors to the user terminal, and when a desired lip color is selected by the user terminal, extracts and presents an eye shadow color and a blusher color corresponding to the selected lip color based on a pre-established database; A virtual makeup solution providing system using generative AI characterized by further including
  9. In a method for providing a makeup solution executed on a makeup service providing server, Step of receiving a face image from a user terminal; A step of analyzing and classifying the above face image using a pre-established feature extraction model to output face feature information; A step of inputting the above facial feature information into a pre-established LLM (Large Language Model)-based Generative AI (Generative Artificial Intelligence) to extract a makeup color corresponding to the above facial feature information; and A step of outputting a virtual makeup image using a diffusion-based generative AI that implants the extracted makeup color into the face image; A method for providing a virtual makeup solution using generative AI including
  10. In Article 9, The above LLM-based generative AI is, A method for providing a virtual makeup solution using a generative AI characterized by being a generative AI trained by instruction tuning using a dataset of facial feature information and makeup colors.

Description

Virtual Makeup Solution Providing System Using Generative Artificial Intelligence The present invention relates to a system for providing a virtual makeup solution using generative AI, and provides a system capable of extracting a makeup color optimal for facial feature information and synthesizing the makeup color onto a face image using diffusion-based generative AI to provide a makeup image. Modern society is an era of advanced technology convergence, with various attempts being made to captivate customers through innovative and smart services incorporating IT. Technological innovation in the beauty industry is also accelerating, and services and products utilizing artificial intelligence are being launched. In particular, as the Millennial generation, which prioritizes personal preferences, rapidly emerges as the primary consumer base for the beauty industry, the sector is presenting more clear and specific personalized beauty solutions through cutting-edge AI technology. Unlike the past, when experiences in the beauty industry were limited to simple promotional methods, the approach is evolving by providing customized product recommendations through facial image analysis. This signifies that new technology has now become a crucial medium for assisting buyers' purchasing behavior and providing customized services to consumers. At this time, a method for performing virtual makeup using artificial intelligence has been researched and developed. In this regard, prior art Korean Published Patent No. 2019-0116052 (published on October 14, 2019) and Korean Registered Patent No. 10-2515436 (published on March 29, 2023) each disclose a configuration in which facial landmarks are extracted using a pre-established deep learning model, personal color is diagnosed using the facial landmarks, and then virtual makeup is performed; and a configuration in which a facial image is acquired, the face is divided into parts to generate information for each part, makeup products for each part are extracted, and then applied to the areas for each part to synthesize them based on GAN. However, the former diagnoses personal color based solely on facial features and does not recommend optimal makeup colors by considering skin tone, eye color, or lip color. In the latter case, since color and face are synthesized using GAN, the face may differ from the original during the synthesis process, potentially leading to a sense of uncanny discrepancy or the uncanny valley. Therefore, research and development of a virtual makeup solution are required that extracts optimal makeup colors by considering not only facial shape but also color, and enables the extracted makeup colors to be synthesized naturally without any sense of uncanny discrepancy with the original. FIG. 1 is a diagram illustrating a virtual makeup solution providing system using generative AI according to an embodiment of the present invention. Figure 2 is a block diagram illustrating a makeup service providing server included in the system of Figure 1. FIGS. 3 and FIGS. 4 are drawings for explaining an embodiment in which a virtual makeup service according to an embodiment of the present invention is implemented. FIG. 5 is an operation flowchart illustrating a method for providing a virtual makeup service according to an embodiment of the present invention. Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals. Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them. Furthermore, when a part is described as "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components, and it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Terms such as “about,” “substantially,” etc., used throughout the specification, are used to mean at or near the stated value when inherent manufacturing and material tolerances are presented in the stated meaning, and are used to prevent unscrupulous infringers from unfairly exploiting the disclosure in which precise or absolute values are mentioned to aid in understanding the invention. Terms such as “step” or “step of” used throughout the specification of the invention do not mean