Search

KR-102962373-B1 - The Method, Service Server, And Computer-Readable Storage Medium That Automatic Creation Of Character Data Used In XR Glass Education

KR102962373B1KR 102962373 B1KR102962373 B1KR 102962373B1KR-102962373-B1

Abstract

The present invention relates to a method for automatically generating character data used in XR glass education, a service server, and a computer-readable storage medium, more specifically, to a method for automatically generating character data used in XR glass education, a service server, and a computer-readable storage medium, wherein the service server recognizes the situation of a character through a character image received from an administrator terminal and generates a first text describing the character, analyzes the character image to determine the difficulty level, extracts object information regarding the character image through a character object derivation model of a classification corresponding to the difficulty level, and analyzes the similarity between a plurality of characters through feature information of the character image and the feature vector of the character's second text to generate a character sequence in which the most different character images exist in adjacent order.

Inventors

  • 조진수
  • 임태훈
  • 우명진

Assignees

  • 가천대학교 산학협력단

Dates

Publication Date
20260507
Application Date
20240329

Claims (12)

  1. A method for automatically generating character data used in XR glasses education performed on a service server comprising one or more processors and one or more memories, wherein A character receiving step of receiving a character image including a character design from an administrator terminal; A character situation recognition step that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction step that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation step of inputting the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation step that generates a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above character situation recognition step is, An object region extraction step of inputting the above character image into a deep learning-based first artificial intelligence model to extract one or more object regions from the character; A keyword extraction step of inputting one or more object regions into a deep learning-based second artificial intelligence model to extract one or more core keywords from the corresponding character; and A method for automatically generating character data used in XR glasses education, comprising: a first text generation step of inputting one or more core keywords and a character image into an internal or external multimodal giant language model to generate a first text related to the character and including the core keywords.
  2. delete
  3. A method for automatically generating character data used in XR glasses education performed on a service server comprising one or more processors and one or more memories, wherein A character receiving step of receiving a character image including a character design from an administrator terminal; A character situation recognition step that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction step that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation step of inputting the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation step for generating a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above-mentioned object extraction step by difficulty level is, An object extraction step of inputting the above character image into a deep learning-based first artificial intelligence model to extract one or more object regions from the character and deriving the number of one or more objects; A color recognition step that recognizes pixel-by-pixel colors in the above character image and analyzes the diversity of colors included in the character; A line segment extraction step for extracting line segments from the above character image and deriving the total length of the line segments included in the character; A difficulty determination step for determining the difficulty of the character based on the number of objects, color diversity, and total length of line segments of the character image; and A method for automatically generating character data used in XR glasses education, comprising: an object information extraction step of selecting one of a plurality of character object derivation models according to the character difficulty determined in the difficulty determination step above, and inputting a corresponding character image into the selected character object derivation model to extract one or more object information for the corresponding character image.
  4. A method for automatically generating character data used in XR glasses education performed on a service server comprising one or more processors and one or more memories, wherein A character receiving step of receiving a character image including a character design from an administrator terminal; A character situation recognition step that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction step that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation step of inputting the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation step that generates a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above text generation step is, A preliminary second text derivation step of inputting the above one or more object information and the above character image into the above internal or external multimodal giant language model to derive multiple preliminary second texts of different difficulty levels; A feature vector extraction step of inputting the plurality of preliminary second texts into a deep learning-based third artificial intelligence model to extract a feature vector for each of the plurality of preliminary second texts; A step for calculating an average feature vector, which is the average value of all feature vectors of multiple preliminary second texts; and A method for automatically generating character data used in XR glasses education, comprising a similarity verification step for analyzing the cosine similarity between each feature vector and the average feature vector.
  5. A method for automatically generating character data used in XR glasses education performed on a service server comprising one or more processors and one or more memories, wherein A character receiving step of receiving a character image including a character design from an administrator terminal; A character situation recognition step that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction step that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation step of inputting the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation step for generating a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above character sequence generation step is, A first random image selection step of randomly selecting one of a plurality of character images as a random image; A first image similarity calculation step for calculating the cosine similarity between the feature information of the random image above and the feature information of each of the remaining character images among all character images; A first text similarity calculation step for selecting a plurality of pre-set number of candidate character images with the lowest similarity to a random image based on the cosine similarity calculated in the first image similarity calculation step, and calculating cosine similarity for the feature vector of the second text of the random image and the feature vector of the second text of each of the candidate character images; and A method for automatically generating character data used in XR glasses education, comprising: a first character sequence derivation step of deriving a candidate character image having the second text with the lowest similarity to the second text of the random image, and deriving a character image among all character images that is least similar to the random image as a first sequence image, which is a sequence image following the random image.
  6. In claim 5, The above character sequence generation step is, A second image similarity calculation step for calculating the cosine similarity between the feature information of the sequence image and the feature information of each of the remaining character images among the entire character image; A second text similarity calculation step for selecting a plurality of pre-set number of candidate character images with the lowest similarity to the sequence image based on the cosine similarity calculated in the second image similarity calculation step, and calculating cosine similarity for the feature vector of the second text of the sequence image and the feature vector of the second text of each of the candidate character images; and A method for automatically generating character data used in XR glasses education, further comprising: a second character sequence derivation step of deriving a candidate character image having the second text with the lowest similarity to the second text of the sequence image, and deriving one character image among all character images that is least similar to the sequence image.
  7. A service server comprising one or more processors and one or more memories, and performing a method for automatically generating character data used in XR glasses education, A character receiving unit that receives a character image including a character design from an administrator terminal; A character situation recognition unit that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction unit that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation unit that inputs the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation unit that generates a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above-mentioned object extraction unit by difficulty level is, An object extraction step of inputting the above character image into a deep learning-based first artificial intelligence model to extract one or more object regions from the character and deriving the number of one or more objects; A color recognition step that recognizes pixel-by-pixel colors in the above character image and analyzes the diversity of colors included in the character; A line segment extraction step for extracting line segments from the above character image and deriving the total length of the line segments included in the character; A difficulty determination step for determining the difficulty of the character based on the number of objects, color diversity, and total length of line segments of the character image; and A service server that performs an object information extraction step, wherein, according to the difficulty of the character determined in the difficulty determination step above, one of a plurality of character object derivation models for each difficulty level is selected, and a character image is input into the selected character object derivation model to extract one or more object information for the character image.
  8. delete
  9. A service server comprising one or more processors and one or more memories, and performing a method for automatically generating character data used in XR glasses education, A character receiving unit that receives a character image including a character design from an administrator terminal; A character situation recognition unit that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction unit that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation unit that inputs the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation unit that generates a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above character sequence generation unit is, A first random image selection step of randomly selecting one of a plurality of character images as a random image; A first image similarity calculation step for calculating the cosine similarity between the feature information of the random image above and the feature information of each of the remaining character images among all character images; A first text similarity calculation step for selecting a plurality of pre-set number of candidate character images with the lowest similarity to a random image based on the cosine similarity calculated in the first image similarity calculation step, and calculating cosine similarity for the feature vector of the second text of the random image and the feature vector of the second text of each of the candidate character images; and A service server that performs a first character sequence derivation step of deriving a candidate character image having the second text with the lowest similarity to the second text of the random image, and deriving a character image among all character images that is least similar to the random image as a first sequence image, which is a sequence image following the random image.
  10. A computer-readable storage medium for implementing a method for automatically generating character data performed on a service server comprising one or more processors and one or more memories, wherein The above computer-readable storage medium includes computer-executable instructions that cause the service server to perform the following steps, and The steps below are: A character receiving step of receiving a character image including a character design from an administrator terminal; A character situation recognition step that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction step that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation step of inputting the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation step for generating a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above-mentioned object extraction step by difficulty level is, An object extraction step of inputting the above character image into a deep learning-based first artificial intelligence model to extract one or more object regions from the character and deriving the number of one or more objects; A color recognition step that recognizes pixel-by-pixel colors in the above character image and analyzes the diversity of colors included in the character; A line segment extraction step for extracting line segments from the above character image and deriving the total length of the line segments included in the character; A difficulty determination step for determining the difficulty of the character based on the number of objects, color diversity, and total length of line segments of the character image; and A computer-readable storage medium comprising: an object information extraction step of selecting one of a plurality of character object derivation models for each difficulty level according to the difficulty of the character determined in the difficulty determination step above, and inputting a corresponding character image into the selected character object derivation model to extract one or more object information for the corresponding character image.
  11. delete
  12. A computer-readable storage medium for implementing a method for automatically generating character data performed on a service server comprising one or more processors and one or more memories, wherein The above computer-readable storage medium includes computer-executable instructions that cause the service server to perform the following steps, and The steps below are: A character receiving step of receiving a character image including a character design from an administrator terminal; A character situation recognition step that recognizes an object area in a received character image, extracts keywords, and generates a first text describing the corresponding character design; A difficulty-specific object extraction step that analyzes a character image to determine the difficulty level of the character, inputs the character image into a character object extraction model of a classification corresponding to the determined difficulty level, and extracts one or more object information for the character image; A text generation step of inputting the above one or more object information and the above character image into a multimodal large language model inside or outside the service server to generate multiple second texts of different difficulty levels; and A character sequence generation step that generates a character sequence including multiple character images in which the most distinct character images exist in adjacent order through similarity analysis of multiple character images and text for multiple corresponding character images; The above character sequence generation step is, A first random image selection step of randomly selecting one of a plurality of character images as a random image; A first image similarity calculation step for calculating the cosine similarity between the feature information of the random image above and the feature information of each of the remaining character images among all character images; A first text similarity calculation step for selecting a plurality of pre-set number of candidate character images with the lowest similarity to a random image based on the cosine similarity calculated in the first image similarity calculation step, and calculating cosine similarity for the feature vector of the second text of the random image and the feature vector of the second text of each of the candidate character images; and A computer-readable storage medium comprising: a first character sequence derivation step of deriving a candidate character image having the second text with the lowest similarity to the second text of the random image, and deriving a character image among all character images that is least similar to the random image as a first sequence image, which is a sequence image following the random image.

Description

The Method, Service Server, And Computer-Readable Storage Medium That Automatic Creation Of Character Data Used In XR Glass Education The Method, Service Server, And Computer-Readable Storage Medium That Automatic Creation Of Character Data Used In XR Glass Education The present invention relates to a method for automatically generating character data used in XR glass education, a service server, and a computer-readable storage medium, more specifically, to a method for automatically generating character data used in XR glass education, a service server, and a computer-readable storage medium, wherein the service server recognizes the situation of a character through a character image received from an administrator terminal and generates a first text describing the character, analyzes the character image to determine the difficulty level, extracts object information regarding the character image through a character object derivation model of a classification corresponding to the difficulty level, and analyzes the similarity between a plurality of characters through feature information of the character image and the feature vector of the character's second text to generate a character sequence in which the most different character images exist in adjacent order. XR (eXtended Reality) refers to extended reality and is used as a term encompassing AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) technologies. Extended Reality is a third reality that integrates virtual reality—which replicates the real world exactly—augmented reality—which combines virtual objects with the real world—and mixed reality—which fuses the real and virtual worlds. Through eyewear devices known as XR glasses, one can experience extended reality that transcends the boundaries between reality and the virtual. XR glasses represent a concept expanded beyond AR glasses, essentially combining AR glasses with content equipped with new extended reality capabilities. By wearing XR glasses, users can experience extended reality by displaying virtual screens on the lenses. Wearing XR glasses not only allows for the display of a massive screen but also enables users to navigate by projecting a virtual image onto one side of the glasses, or follow along with videos on furniture assembly or cooking methods. XR technology can be utilized in various fields such as culture and arts, medical imaging, and product manufacturing. In particular, because XR technology enables realistic experiences, it can be applied to the field of education. By displaying desired screens through XR glasses, realistic education in various fields can be conducted for learners of all ages, from infants to adults. Meanwhile, education using XR glasses is possible for children or learners with visual impairments including macular holes, tunnel vision, and color blindness, and among them, education can be conducted using character images while wearing XR glasses. Since education is possible by displaying character images suitable for the learner's difficulty level through XR glasses, there is a need for a system that can continuously generate various character images without overlapping. Therefore, there is a need for technology that can automatically and continuously provide new characters for the character images that can be displayed on XR glasses. Prior art in this field includes technology related to an apparatus and method for providing an Extended Reality (XR)-based learning platform, such as Korean Registered Patent No. 10-2299065. However, while the aforementioned prior art suggests a method for providing an Extended Reality (XR)-based learning platform that performs interaction via AR according to the character journey of the original story, it does not disclose at all a method for conducting education by displaying various character images through XR glasses, as previously mentioned; therefore, there is a need for technology to resolve this issue. FIG. 1 schematically illustrates the connection configuration of a service server according to one embodiment of the present invention. FIG. 2 schematically illustrates the connection configuration of a service server according to one embodiment of the present invention. FIG. 3 schematically illustrates the internal configuration of a service server according to one embodiment of the present invention. FIG. 4 schematically illustrates the internal configuration of a character situation recognition unit according to one embodiment of the present invention. FIG. 5 schematically illustrates the process of performing a character situation recognition step according to one embodiment of the present invention. FIG. 6 schematically illustrates the internal configuration of an object extraction unit by difficulty level according to one embodiment of the present invention. FIG. 7 schematically illustrates the process of performing the step of determining the difficulty of a character in the object extraction step