Search

KR-102963970-B1 - Illegal advertising detection apparatus using multimodal learning and method thereof

KR102963970B1KR 102963970 B1KR102963970 B1KR 102963970B1KR-102963970-B1

Abstract

The present invention relates to an apparatus and method for detecting illegal advertisements using multimodal learning, which can prevent damage to legitimate advertisers by rapidly detecting copyright-infringing advertisements that unauthorizedly reproduce or modify advertisement content, as well as modified advertisements containing false or exaggerated content that cause damage to the original advertiser. The invention separates images and text from image-based advertisement content, obtains the degree of similarity to the original advertisement content from an image perspective and a text perspective through a similarity judgment unit trained by a separate learning method for each, maps them to a single data space through a learner trained by a multimodal joint representation method, and determines the similarity between the original advertisement content and the target advertisement content based on distance. This enables rapid and accurate determination of similarity for advertisement content in a complex, unstructured data state, thereby efficiently detecting illegal advertisement content such as reproduction or modification and effectively blocking the illegal use of advertisement content.

Inventors

  • 이상민
  • 오수빈
  • 조민수
  • 김주현
  • 곽소정
  • 김채현

Assignees

  • 광운대학교 산학협력단

Dates

Publication Date
20260512
Application Date
20221130

Claims (10)

  1. An ad preprocessing unit that distinguishes image and text regions in target ad content and generates ad text extracted from normalized ad images and text regions; An image similarity analysis unit that determines and provides similarity weights of normalized advertisement images generated by the advertisement preprocessing unit through a learning unit trained with original advertisement content images; A text similarity analysis unit that determines and provides similarity weights of the advertisement text generated by the advertisement preprocessing unit through a learning unit trained on an interpretation method of importance per word of the original advertisement content text; A multimodal joint analysis unit that maps the similarity weights of the image similarity analysis unit and the text similarity analysis unit into a joint embedding space to determine the similarity of the target advertising content to the original advertising content based on distance, and provides target advertising content information having a similarity greater than or equal to a set standard as similar advertising content information; It includes a false exaggeration analysis unit that provides false exaggeration risk information of text data corresponding to similar advertising content information provided by the multimodal joint analysis unit, through a learning unit trained with a dictionary of allowed and prohibited expressions regarding false and exaggerated advertising, The above advertisement preprocessing unit A text selection and reading unit that distinguishes text areas in target advertising content and original advertising content, reads the characters in the distinguished text area to match and store text data with advertising content information, and also stores information regarding text size and color as feature information to be used as weights for determining importance; An image selection normalization unit that distinguishes image regions from target ad content and original ad content, overlaps them using a patch method to generate a single normalized image of a fixed size, and then saves it by matching it with ad content information; It is characterized by including an ad-specific data management unit that generates ad content information to distinguish between original and each target ad content, and provides image data or text data matched and stored with said ad content information upon request. An illegal advertisement detection device using multimodal learning, characterized in that the text similarity analysis unit and the false exaggeration analysis unit further utilize information regarding visually emphasized important words or phrases by applying feature information including information on text size and color as weights in addition to the advertisement text generated by the advertisement preprocessing unit.
  2. delete
  3. An illegal advertisement detection device using multimodal learning according to claim 1, further comprising an advertisement collection unit that collects new advertisement content generated during a preset period and provides it to the advertisement preprocessing unit.
  4. An illegal advertisement detection device using multimodal learning according to claim 1, characterized in that it includes a result output unit that collects similar advertisement content information from the multimodal joint analysis unit to provide a list of illegal advertisements, and collects false exaggeration risk information regarding similar advertisement content determined by the false exaggeration analysis unit to provide a list of advertisements at risk of false exaggeration.
  5. delete
  6. The illegal advertisement detection device using multimodal learning according to claim 1, wherein the text similarity analysis unit includes a transformer learner trained through a training dataset in which the text of the original advertisement content and the text of the similar advertisement content are labeled, and the transformer learner performs word embedding based on a label to determine whether each word of the input text is a word included in the original text, encodes based on location by reflecting weights according to feature information regarding the size and color of the text, learns attention values and location information indicating bidirectional word associations with each specific word in the sentence through an encoder, and outputs weights in a fully connected manner through a decoder.
  7. An illegal advertisement detection device using multimodal learning according to claim 1, wherein the multimodal joint analysis unit comprises a learning unit that constructs a neural network for joint representation of similarity weights of the image similarity analysis unit and the text similarity analysis unit and performs learning to vectorize and map two different modalities in one space, and determines similarity based on the distance of the location obtained for the target advertisement using the location of the original advertisement obtained through the learning unit as an anchor.
  8. An illegal advertisement detection device using multimodal learning according to claim 1, wherein the false exaggeration analysis unit includes a transformer learner in which expression learning is performed through a dictionary of allowed and prohibited expressions for false and exaggerated advertisements, and the transformer learner includes a word embedding configuration in which the inclusion of false expressions, the level of inclusion of exaggerated advertisement expression words, and the level of sentiment analysis of expressions are all considered.
  9. As a method for detecting illegal advertisements using an illegal advertisement detection device, A step of distinguishing image regions and text regions from original advertising content and similar advertising content, labeling generated image data and text data, and training learners of the image similarity analysis unit and the text similarity analysis unit; A step of training a learner of a false or exaggerated advertising analysis unit to determine the risk of false or exaggerated advertising expressions being included in text data using a dictionary of allowed and prohibited expressions regarding false or exaggerated advertising; A step of collecting target ad content generated during a preset period, separating image areas and text areas, and generating image data and text data; A step of calculating original advertising content similarity weights for image data and text data of the above-mentioned target advertising content through image similarity analysis and text similarity analysis, respectively; A step of mapping the calculated different types of similarity weights into a single space, selecting similar target advertising content based on distance according to the spatial location of the original advertising content, and providing similar advertising content information; The method includes the step of calculating, through the false or exaggerated analysis unit, the risk of including false or exaggerated advertising expressions based on the text data of each similar advertising content corresponding to the selected similar advertising content information. The step of generating the above text data includes a step of generating additional feature information based on the size and color of the text in addition to the text data. A method for detecting illegal advertisements using multimodal learning, characterized in that the step of calculating the similarity weights of the advertisement content through image similarity analysis and text similarity analysis, respectively, further includes a step of utilizing information on visually emphasized important words or phrases by reflecting feature information based on the size and color of the text in addition to text data as a weight in the input during the text similarity analysis.
  10. delete

Description

Illegal advertising detection apparatus using multimodal learning and method thereof The present invention relates to an apparatus and method for detecting illegal advertisements using multimodal learning, and more specifically, to an apparatus and method for detecting illegal advertisements using multimodal learning that can prevent damage to legitimate advertisers by rapidly detecting advertisements that cause damage to the original advertiser, such as copyright-infringing advertisements that unauthorizedly reproduce or similarly modify advertisement content, or modified advertisements similar to copyright-infringing advertisements that contain false or exaggerated content. As online product sales become more active, online advertising content for product sales is surging. Such online advertising content includes not only pure advertising content intended simply to provide information about products, but also content provided through sales pages that offer various information about the products for sale. In other words, because access to markets or web pages for product sales is free, product descriptions on sales pages are used as advertising content. Online advertising content containing product descriptions and advertisements features various images and text appropriately arranged according to the content creator's intent; therefore, the arrangement of such advertising content as well as the text and images included in it are all subject to copyright. Furthermore, due to the nature of advertising content, the content must not contain any content that constitutes false or exaggerated advertising prohibited by law. However, recent online product sales involve multiple sellers competing with similar products, and the reality is that even identical products are being sold competitively by different sellers using their respective unique advertising content. Therefore, such advertising content works are frequently reproduced or modified and used within a short period of time, and there are even cases where false or exaggerated advertising expressions are added to the reproduced or modified advertising content in an attempt to increase the sales volume of the products being sold. The reproduction or modification of such advertising content not only infringes upon the copyrights of the creators of the advertising content or legitimate advertisers but also constitutes unfair competition and must be eradicated. Furthermore, modifying unauthorized content to add false or exaggerated advertising content constitutes deception of consumers, causes serious business damage to legitimate content creators or advertisers, and lowers the credibility of the product itself. However, due to the nature of advertising content in which text and images are mixed in an unstructured manner, it is difficult to easily find infringing advertising content that copies or modifies legitimate advertising content among the vast amount of online advertising content, and furthermore, it is also difficult to detect the risk of false or exaggerated advertising. In particular, while it is somewhat possible to identify web pages containing images similar to a specific image using current similar image detection technology, it is difficult to detect similar advertising content because advertising content, which consists of a mixture of images and text, is generated as a single large image; therefore, merely changing the text size or the positions of text and images causes the images to exceed the similarity range. Nevertheless, cases of copying or modifying such advertising content, as well as cases of applying false or exaggerated advertising, are increasing, making it urgent to find new methods to eradicate them. FIG. 1 is a configuration diagram of an illegal advertisement detection device according to an embodiment of the present invention. FIG. 2 is a conceptual configuration diagram of an advertisement preprocessing unit according to an embodiment of the present invention. FIG. 3 is a configuration and conceptual diagram of a multimodal learning method according to an embodiment of the present invention. FIG. 4 is a configuration and conceptual diagram of the analysis of false and exaggerated advertisements and the provision of results according to an embodiment of the present invention. FIG. 5 is a configuration diagram of the learning unit of the text similarity analysis unit according to an embodiment of the present invention. FIG. 6 is a configuration diagram of the learning unit of the false exaggeration analysis unit according to an embodiment of the present invention. It should be noted that the technical terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. Furthermore, unless specifically defined otherwise in this invention, the technical terms used in this invention should be interpreted in the sense generally understood by those skilled in the art to which this