KR-20260067939-A - APPARATUS FOR PROVIDING INTERACTIVE SIGN LANGUAGE GUIDANCE SERVICE USING LARGE LANGUAGE MODELS

KR20260067939AKR 20260067939 AKR20260067939 AKR 20260067939AKR-20260067939-A

Abstract

The present invention relates to an interactive sign language guidance service providing device utilizing a large language model. According to one aspect of the present invention, the interactive sign language guidance service providing device utilizing a large language model receives a sign language question composed of sign language words, converts the sign language question into a sentence-form Korean question through a pre-trained sign language-Korean large language model, generates a Korean answer to the Korean question through a search augmentation generation-based answer large language model, and converts the Korean answer into a sign language answer composed of sign language words through a sign language-Korean large language model and outputs it.

Inventors

박진선
허석용
김진완
전민수
하성헌

Assignees

부산대학교 산학협력단

Dates

Publication Date: 20260513
Application Date: 20241206
Priority Date: 20241105

Claims (6)

A device for providing an interactive sign language guidance service utilizing a large language model, comprising: an answer module that receives a sign language question composed of sign language words, converts the sign language question into a sentence-form Korean question through a pre-trained sign language-Korean large language model, generates a Korean answer to the Korean question through a search augmentation generation-based answer large language model, and converts the Korean answer into a sign language answer composed of sign language words through the sign language-Korean large language model and outputs it.
In paragraph 1, An interactive sign language guidance service providing device utilizing a large language model, further comprising: an answer refinement module that receives a sign language answer output from the above-mentioned answer module and outputs a refined final sign language answer by replacing words included in the sign language answer with previously stored sign language words based on the similarity between the sign language answer and previously stored sign language words according to words and context.
In paragraph 2, The above answer refinement module is A sign language database that stores sign language words and sentences containing each sign language word; A candidate sign language word selection unit that receives a sign language answer output from the above answer module, calculates the similarity between each word included in the above sign language answer and each sign language word stored in the above sign language database, and selects the sign language words as candidate sign language words for each word included in the above sign language answer according to the calculated similarity. A device for providing interactive sign language guidance services utilizing a large language model.
In paragraph 3, The above candidate sign language word selection department A first similarity is calculated using the cosine similarity between the embedding vector of each word included in the above sign language answer and the embedding vector of each sign language word stored in the above sign language database, and A second similarity is calculated using the cosine similarity between the context embedding vector of the sentence containing each word in the above sign language answer and the context embedding vector of the sentence containing each sign language word stored in the above sign language database, Calculating the final similarity of the sign language words for each word by weighting the first similarity and the second similarity with preset weights. A device for providing interactive sign language guidance services utilizing a large language model.
In paragraph 3, The above candidate sign language word selection department If the similarity calculated for each word included in the sign language response exceeds a preset threshold, the sign language word is selected as a candidate sign language word for each word included in the sign language response. A device for providing interactive sign language guidance services utilizing a large language model.
In paragraph 5, The above candidate sign language word selection department Calculate an evaluation score for candidate sign language words for each word based on the similarity between each word included in the above sign language answer and the candidate sign language words selected for each word included in the above sign language answer, and the frequency of the candidate sign language words in the above sign language database. Selecting the candidate sign language word with the highest evaluation score based on the evaluation scores of candidate sign language words for each word, and selecting it as the optimal sign language word for each word included in the above sign language answer. A device for providing interactive sign language guidance services utilizing a large language model.

Description

Apparatus for Providing Interactive Sign Language Guidance Service Using Large Language Models The present invention relates to a technology for performing translation between sign language and Korean. Current Sign Language-Korean translation technology relies on machine translation and image recognition technologies. In Sign Language, the same word can be expressed in various ways depending on regional and cultural differences. Existing Sign Language-Korean translation technologies have limitations in naturally converting Sign Language expressions into Korean because they fail to consider the natural flow of Korean sentences or the nuances of Sign Language users in various situations. Furthermore, existing sign language-Korean translation technology merely provides unilateral translation, such as simply translating sign language into Korean or Korean into sign language, and fails to provide two-way responses. Search Augmented Generative (RAG) technology is a technology that generates a response by searching for information about a question and reflecting the search results in a large language model. It is used in chatbot services and the like, but there are no cases of it being applied to sign language translation. In particular, existing sign language guidance services are limited to providing one-way sign language-to-Korean translation, so there is a need to develop technology that provides customized responses to sign language. FIG. 1 is a block diagram of an interactive sign language guidance service providing device utilizing a large language model according to an embodiment of the present invention. FIG. 2 is a flowchart of the operation of an interactive sign language guidance service providing device utilizing a large language model according to an embodiment of the present invention. The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the claims. Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The present invention relates to a technology for providing a sign language guidance service, and is characterized in that it provides a sign language guidance service capable of two-way communication of sign language question recognition and response. These technical features can be achieved by a configuration that uses a large language model to translate a sign language question into a Korean sentence, generates a Korean answer to the question translated into a Korean sentence using search augmentation technology, and translates the Korean answer back into sign language using the large language model. In addition, the present invention can provide a natural translation by not immediately outputting the result of translating a Korean answer into sign language, but by refining and outputting it based on the similarity in words and context with sign language words stored in a sign language database. Specifically, referring to FIG. 1, an interactive sign language guidance service providing device (100) utilizing a large language model according to an embodiment of the present invention may be configured to include an answer module (110) and an answer refinement module (120). The answer module (110) may receive a sign language question consisting of sign language words, convert the sign language question into a sentence-form Korean question through a pre-trained sign language-Korean big language model, generate a Korean answer to the Korean question through a search augmentation generation-based answer big language model, and convert the Korean answer into a sign language answer consisting of sign language words through the sign language-Korean big language model and output it. The answer module (110) may be configured to include a sign language-Korean large language model (111), a search augmentation generation-based answer large language model (112), a Korean question conversion unit (113), an answer generation unit (114), and a sign language answer conversion unit (115). The sign language-Korean giant language model (111) may be pre-trained to receive an array of sign language words and convert it into a Korean sentence, or receive a Korean sentence and convert it into an array of sign language words. The search augmentation generation-based answer-large language model (112) may include a knowledge databa