US-20260127379-A1 - METHOD, APPARATUS AND COMPUTER-READABLE MEDIUM FOR IDENTIFYING MICRO BIASED TEXT WITHIN OPEN CORPORA AND GENERATING RESPONSE TO IDENTIFIED MICRO BIASED TEXT

US20260127379A1US 20260127379 A1US20260127379 A1US 20260127379A1US-20260127379-A1

Abstract

Provided is a method for identifying subtly biased texts within open corpora, which includes: a natural language sentence collecting step of collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining step of comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; and a bias determining step of determining the bias candidate dataset as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models.

Inventors

Jai Eun KIM
Sang Min Park

Assignees

SALTLUX INC.

Dates

Publication Date: 20260507
Application Date: 20241219
Priority Date: 20241107

Claims (20)

1 . A method implemented by a computing device including at least one processor and at least one memory for storing instructions executable by the processor to identify subtly biased texts within open corpora, the method comprising: a natural language sentence collecting step of collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining step of comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; and a bias determining step of determining the bias candidate dataset as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models.
2 . The method of claim 1 , wherein the bias determining step includes using, as the multiple large-scale language models, heterogeneous large-scale language models having different structures and training mechanisms.
3 . The method of claim 1 , wherein the bias determining step includes: when each of the multiple large-scale language models performs bias determination on one bias candidate dataset subject to the bias determination and when all of the large-scale language models determine the one bias candidate dataset as a biased dataset, classifying the one bias candidate dataset as the surficial bias dataset.
4 . The method of claim 1 , wherein the bias determining step includes: when each of the multiple large-scale language models performs bias determination on one bias candidate dataset subject to the bias determination and when all of the large-scale language models determine the one bias candidate dataset as a unbiased dataset, classifying the one bias candidate dataset as the non-bias dataset.
5 . The method of claim 1 , wherein the bias determining step includes: when each of the multiple large-scale language models performs bias determination on one bias candidate dataset subject to the bias determination and when at least one of the large-scale language models determines the one bias candidate dataset as a biased dataset, classifying the one bias candidate dataset as the subtle bias dataset.
6 . The method of claim 5 , further comprising: a determination criterion querying step, when the one bias candidate dataset is classified as the subtle bias dataset by performing the bias determining step, of querying a determination criterion to the large-scale language model having obtained the classification results as the subtle bias dataset; a response validity determining step of, when a response returned by the large-scale language model is present by performing the determination criterion querying step, determining validity of the returned response; and a learning dataset storing step of classifying the subtle bias dataset, which is obtained as the valid response in the response validity determining step, as a learning dataset for self-learning to store the classified the subtle bias dataset in a learning database.
7 . The method of claim 6 , wherein the response validity determining step includes determining the response as a valid response when a factor containing at least one of keywords and ideas containing social and ethical issues is obtained from the response returned by the large-scale language model.
8 . The method of claim 6 , further comprising: a self-learning step, after the bias determining step, of performing self-learning by providing subtle bias-related learning data to the multiple large-scale language models by using the learning data sets stored in the learning database.
9 . The method of claim 1 , wherein the bias determining step includes: when each of the multiple large-scale language models performs bias determination on one bias candidate dataset subject to the bias determination and when the multiple large-scale language models provide different results on the bias determination, classifying the one bias candidate dataset as the subtle bias dataset when there are a majority of determinations on ‘biased’ and classifying the one bias candidate dataset as the non-bias dataset when there are a majority of determinations on ‘non-bias’, based on a majority rule.
10 . A method implemented by a computing device including one or more processors and one or more memories for storing instructions executable in the processors to identify subtly biased text within open corpora and generate a response to the identified subtly biased text, the method comprising: a natural language sentence collecting step of collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining step of comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; a bias determining step of determining the bias candidate dataset as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models; and a response answer generating step of generating a response answer for the natural language sentence identified as the surficial bias dataset and the subtle bias dataset in the bias determining step so as to generate a response answer in which bias is mitigated or removed.
11 . The method of claim 10 , wherein the response answer generating step includes generating the response answer to the natural language sentence having identified bias in a format that includes at least one of a summary format and a detailed description format.
12 . The method of claim 10 , wherein the bias determining step includes: a bias determination result receiving step of receiving bias determination results for the one bias candidate dataset from the multiple large-scale language models; a bias score calculating step of calculating a bias score for the one bias candidate dataset based on the majority rule by aggregating the received bias determination results; and a bias intensity defining step of defining a bias intensity for the one bias candidate dataset based on the calculated bias score.
13 . The method of claim 12 , wherein a trust level is defined for each of the large-scale language models based on a trust level management model preset for the multiple large-scale language models, and the bias score calculating step includes calculating the bias score by giving a highest weight to the bias determination result provided by the large-scale language model defined with a highest trust level.
14 . The method of claim 12 , wherein the bias intensity defining step includes, based on a preset threshold bias score, defining the one bias candidate dataset as a first bias level when the calculated bias score is less than the threshold bias score, and defining the one bias candidate dataset as a second bias level when the calculated bias score is greater than or equal to the threshold bias score.
15 . The method of claim 14 , wherein the response answer generating step, when the bias intensity defined for the one bias candidate dataset is the first bias level, includes generating a first response answer, which is a response answer composed of correction information that corrects the bias.
16 . The method of claim 14 , wherein the response answer generating step, when the bias intensity defined for the one bias candidate dataset is the second bias level, includes generating a second response answer, which is a response answer composed of warning information that warns of the bias, together with the correction information that corrects the bias.
17 . An apparatus implemented by a computing device including one or more processors and one or more memories for storing instructions executable in the processors to identify subtly biased texts within open corpora, the apparatus comprising: a natural language sentence collecting unit for collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining unit for comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; and a bias determining unit for determining the bias candidate dataset, which is obtained from the bias candidate dataset obtaining unit, as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models.
18 . An apparatus implemented by a computing device including one or more processors and one or more memories for storing instructions executable in the processors to identify subtly biased texts within open corpora and generate a response to the identified subtly biased texts, the apparatus, the apparatus comprising: a natural language sentence collecting unit for collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining unit for comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; a bias determining unit for determining the bias candidate dataset, which is obtained from the bias candidate dataset obtaining unit, as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models; and a response answer generating unit for generating a response answer for the natural language sentence identified as the surficial bias dataset and the subtle bias dataset in the bias determining unit so as to generate a response answer in which bias is mitigated or removed.
19 . A computer-readable recording medium storing instructions for allowing a computing device to perform steps comprising: a natural language sentence collecting step of collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining step of comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; and a bias determining step of determining the bias candidate dataset as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models.
20 . A computer-readable recording medium storing instructions for allowing a computing device to perform steps comprising: a natural language sentence collecting step of collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining step of comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset including seed bias data expected to be socially and ethically biased and derived bias data expanded from the seed bias data; a bias determining step of determining the bias candidate dataset as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models; and a response answer generating step of generating a response answer for the natural language sentence identified as the surficial bias dataset and the subtle bias dataset in the bias determining step so as to generate a response answer in which bias is mitigated or removed.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for identifying subtly biased text within open corpora and generating a response to the identified subtly biased text, and more particularly, to a technology of identifying texts having latent biases that are not explicitly apparent in the open corpora to establish data on subtle bias and performing learning on the established data, thereby detecting wrong biases of large-scale language models, so as to solve degradation of reliability and performance for the large-scale language models. 2. Description of the Related Art Recently, along with development of artificial intelligence technology, natural language processing technology has been also rapidly developing. In particular, performance of natural language tasks has been dramatically improved with the release of translation machines to which self-attention and multi-head attention technologies among various models for neural machine translation are applied. The BERT model, which uses only the encoder block of a translation machine, has greatly contributed to the revival of deep learning technology for processing natural languages, and the GPT3 model, which uses only the decoder block, has opened a new chapter in natural language generation by artificial intelligence through learning on huge corpora. However, the development of artificial intelligence technology (that is, large-scale language models) in the field of natural language processing has faced ethical issues on artificial intelligence, such as the ‘ILuda (Luda.ai) controversy’. In other words, the artificial intelligence having learned various hate speech, personal information and politically/ethically biased information present in data input for learning mechanically may provide biased predictions and results without any sense of guilt, and this problem may not only cause a fatal weakness in reliability on large-scale language models, but also cause a major limitation to commercialization. Accordingly, Korean Unexamined Patent publication No. 10-2023-0075890 (Language model output device and method with removed bias), proposes a language model output technology for determining biases and removing generated bias information by removing modules to remove bias through human intervention in the process of deep learning. Meanwhile, the above-mentioned related art is technology that removes biases by comparing and reviewing biased information generated by large-scale language models with main information constructed through human intervention. Accordingly, there is a high possibility of human error due to the human intervention, and it is difficult to identify subtle biases that are not explicitly apparent even when explicitly apparent biases are identified, and thus there may be a limitation in solving deterioration of reliability on large-scale language models. SUMMARY OF THE INVENTION In this regard, a primary object of the present invention is to provide technology for allowing a large-scale language model to identify texts having subtle biases (or latent biases), which are not explicitly apparent in open corpora, corresponding to biases that cannot be easily detected by the large-scale language model. In addition, a secondary object of the present invention is to provide technology for generating and providing unbiased response answers when a large-scale language model receives a question about an identified subtly biased text, so that fairness, ethics and reliability of the large-scale language model may be promoted. In order to achieve the above objects, a method implemented by a computing device including one or more processors and one or more memories for storing instructions executable in the processors according to one embodiment of the present invention to identify subtly biased texts within open corpora includes: a natural language sentence collecting step of collecting natural language sentences from text-based datasets including open conversation datasets and web corpus datasets; a bias candidate dataset obtaining step of comparing semantic vectors between words composing the natural language sentences by using a word-embedding model from the collected natural language sentences, thereby obtaining a bias candidate dataset expected to be socially and ethically biased; and a bias determining step of determining the bias candidate dataset as one of a surficial bias dataset, a subtle bias dataset, and a non-bias dataset by using multiple large-scale language models. The bias determining step may include using, as the multiple large-scale language models, heterogeneous large-scale language models having different structures and training mechanisms. In addition, when each of the multiple large-scale language models performs bias determination on one bias candidate dataset subject to the bias determination and when all of the large-scale language models determine the one bias candidate dataset as a