JP-7856222-B1 - Question generation method, knowledge extraction method, question answering system construction method, question generation program, knowledge extraction program, question generation system, and knowledge extraction system

JP7856222B1JP 7856222 B1JP7856222 B1JP 7856222B1JP-7856222-B1

Abstract

The question generation method according to the present invention includes: a theme information receiving step for receiving theme information to generate a question; an extraction step for extracting local information from the received theme information; a supplemental information generation step for generating supplemental information for the extracted local information; a text acquisition step for acquiring text generated using a large-scale language model that learns local information as features based on a prompt including theme information, local information, and supplemental information; and a determination step for determining whether or not to set a text as a question related to a theme, based on the degree of content overlap and/or theme relevance.

Inventors

原田洋平
丸山純平

Assignees

ＪＦＥスチール株式会社

Dates

Publication Date: 20260511
Application Date: 20250722
Priority Date: 20240827

Claims (16)

A method for generating questions performed by a computer, A theme information reception step that accepts theme information input for generating questions, From the received theme information, there is an extraction step to extract local information, A supplementary information generation step for generating supplementary information for the extracted local information, A text acquisition step involves acquiring text generated using a large-scale language model that learns the local information as a feature, based on a prompt including the theme information, the local information, and the supplementary information; A question refinement step in which, with respect to the aforementioned text, a question is refined to determine whether or not to set the text as a question relating to the theme information, based on the degree of overlap in content with other texts generated prior to the text being evaluated, and/or the degree of theme relevance to the theme information; A method for generating questions that includes this.
If, in the question review step, it is determined that the question should not be set as a question relating to the theme information, the text obtained in the text acquisition step is set as a supplementary prompt that includes the text as an inappropriate example and provides instructions to encourage improvement of the output. Based on the aforementioned prompt and the supplementary prompt, the text acquisition step is executed again. The question generation method according to claim 1.
As a guideline for questions regarding thematic information, it further includes a topic setting step to define topics for questions related to thematic information. In the text acquisition step, information related to the topic of the question that was set is added to the prompt. The question generation method according to claim 1 or 2.
The aforementioned topic setting step sets the topic of the question using a large-scale language model that is generated by learning local information as features, based on prompts that include theme information and prompt the generation of a question topic. The question generation method according to claim 3.
The aforementioned question review step compares the similarity between the information obtained by vectorizing the text and the information obtained by vectorizing other texts generated before the text being evaluated, in order to determine whether or not there is any overlap in content. The question generation method according to claim 1.
The aforementioned question review step determines whether the text is related to the set theme based on the relationship between the information obtained by vectorizing the text and the information obtained by vectorizing the theme information. The question generation method according to claim 1.
The extraction step involves extracting local information from the theme information by referring to a database that has been pre-stored with local terms and sentences containing those local terms. The question generation method according to claim 1.
The extraction step involves extracting the local information by referring to a database that stores information obtained by pre-vectorizing local terms and sentences containing those local terms, and comparing the similarity between that information and the information obtained by vectorizing the theme information. The question generation method according to claim 1.
A method of knowledge extraction performed by a computer, A theme information reception step that accepts theme information input for generating questions, The first extraction step involves extracting local information from the received theme information, A first supplementary information generation step in which first supplementary information is generated with respect to the extracted first local information, A text acquisition step in which text is acquired using a large-scale language model that is generated by learning the local information as a feature, based on a prompt including the theme information, the first local information, and the first supplementary information, A question refinement step in which, with respect to the aforementioned text, a question is refined to determine whether or not to set the text as a question relating to the theme information, based on the degree of overlap in content with other texts generated prior to the text being evaluated, and/or the degree of theme relevance to the theme information; A question output step that outputs a question that has been determined to be set as the question by the question review step, The response reception step involves receiving response information from respondents, A second extraction step is performed to extract second local information from the aforementioned response information, A second supplemental information generation step is performed to generate second supplemental information with respect to the extracted second local information, A knowledge information generation step that generates knowledge information by converting information from a series of processes, including the aforementioned question, the aforementioned answer information, second local information, and second supplementary information, into a set format, A knowledge extraction method that includes this.
The method further includes an update step to update the prompt by adding the response information, the second local information, and the second supplementary information to the prompt, The document acquisition step, the question review step, the question output step, the answer reception step, the second extraction step, and the second supplemental information generation step are repeatedly executed based on the updated prompt. The knowledge extraction method according to claim 9.
In the knowledge information generation step, the knowledge information is converted into a knowledge graph format and a text format to generate the knowledge information. The knowledge extraction method according to claim 9 or 10.
A step of generating knowledge information converted into knowledge graph format and text format using the knowledge extraction method described in claim 11 , The steps include saving the generated knowledge information to a database, A Retrieval Augmented Generation (RAG) environment is constructed as a question answering system, in which the knowledge information stored in the database is provided to a large-scale language model, enabling the retrieval of the knowledge information from the database and the generation of a response . Method for building a question-answering system.
A theme information reception step that accepts theme information input for generating questions, The extraction step involves referencing the memory unit to extract local information from the received theme information, A supplementary information generation step for generating supplementary information for the extracted local information, A text acquisition step involves acquiring text generated using a large-scale language model that learns the local information as a feature, based on a prompt including the theme information, the local information, and the supplementary information; A question refinement step in which, with respect to the aforementioned text, a question is refined to determine whether or not to set the text as a question relating to the theme information, based on the degree of overlap in content with other texts generated prior to the text being evaluated, and/or the degree of theme relevance to the theme information; A question generation program that causes a computer to execute a question.
A theme information reception step that accepts theme information input for generating questions, A first extraction step involves extracting first local information by referring to the memory unit from the received theme information, A first supplementary information generation step in which first supplementary information is generated with respect to the extracted first local information, A text acquisition step in which text is acquired using a large-scale language model that is generated by learning the local information as a feature, based on a prompt including the theme information, the first local information, and the first supplementary information, A question refinement step in which, with respect to the aforementioned text, a question is refined to determine whether or not to set the text as a question relating to the theme information, based on the degree of overlap in content with other texts generated prior to the text being evaluated, and/or the degree of theme relevance to the theme information; A question output step that outputs the questions set by the question review step, The response reception step involves receiving response information from respondents, A second extraction step is performed to extract second local information from the aforementioned response information, A second supplemental information generation step is performed to generate second supplemental information with respect to the extracted second local information, A knowledge information generation step that generates knowledge information by converting information from a series of processes, including the aforementioned question, the aforementioned answer information, second local information, and second supplementary information, into a set format, A knowledge extraction program that causes a computer to execute a command.
A theme information receiving unit that accepts theme information input for generating questions, An extraction unit extracts local information by referencing the memory unit from the received theme information, A supplementary information generation unit generates supplementary information for the extracted local information, A text acquisition unit that acquires text generated using a large-scale language model that learns the local information as a feature based on a prompt including the theme information, the local information and the supplementary information, A question review unit determines whether or not to set the aforementioned text as a question relating to the theme information, based on the degree of overlap in content with other texts generated prior to the text being evaluated, and/or the degree of theme relevance to the theme information. A question generation system equipped with the following features.
A theme information receiving unit that accepts theme information input for generating questions, A first extraction unit extracts first local information by referring to the memory unit from the received theme information, A first supplementary information generation unit generates first supplementary information with respect to the extracted first local information, A text acquisition unit acquires text generated using a large-scale language model that learns the local information as features based on a prompt including the theme information, the first local information, and the first supplementary information. A question review unit determines whether or not to set the aforementioned text as a question relating to the theme information, based on the degree of overlap in content with other texts generated prior to the text being evaluated, and/or the degree of theme relevance to the theme information. A question output unit that outputs a question that the question review unit has determined to set as the question, The response reception department receives response information from respondents, A second extraction unit extracts second local information from the aforementioned response information, A second supplemental information generation unit generates second supplemental information for the extracted second local information, A knowledge information generation unit generates knowledge information by converting information from a series of processes, including the aforementioned question, the aforementioned answer information, second local information, and second supplementary information, into a set format. A knowledge extraction system equipped with the following features.

Description

This invention relates to a question generation method, a knowledge extraction method, a question answering system construction method, a question generation program, a knowledge extraction program, a question generation system, and a knowledge extraction system. In recent years, the transfer of skilled skills in the manufacturing industry has become a problem. In particular, the knowledge of skilled workers, especially undocumented tacit knowledge, urgently needs to be formally preserved in some form. To formalize human tacit knowledge, questionnaires and interviews are commonly used. As a technique for formalizing tacit knowledge, methods for conducting interviews suitable for extracting tacit knowledge are known, such as those described in Non-Patent Document 1. Rie Yabutani, "Effectiveness of the 'Functional Approach to Tacit Knowledge Extraction' Method," Artificial Intelligence Society of Japan Research Meeting Proceedings, SIG-KST-043-03 (2023-03-23) Figure 1 is a block diagram showing the schematic configuration of a knowledge processing system according to one embodiment of the present invention.Figure 2 is a block diagram showing the configuration of a knowledge extraction device included in a knowledge processing system according to one embodiment of the present invention.Figure 3 is a block diagram showing the configuration of a text generation device included in a knowledge processing system according to one embodiment of the present invention.Figure 4 is a sequence diagram illustrating the flow of knowledge extraction processing according to one embodiment of the present invention.Figure 5 is a diagram (part 1) illustrating an example of interaction with a respondent during knowledge extraction according to one embodiment of the present invention.Figure 6 is a diagram (part 2) illustrating an example of interaction with the respondent during knowledge extraction according to one embodiment of the present invention.Figure 7 is a sequence diagram illustrating the flow of the knowledge extraction process according to Modification 1 of the present invention.Figure 8 is a sequence diagram illustrating the flow of the knowledge extraction process according to Modification 2 of the present invention.Figure 9 shows the tacit knowledge information, which is not documented, extracted from skilled technicians responsible for equipment maintenance at a steel mill using the knowledge extraction process of the present invention, as both a text-format summary and a knowledge graph.Figure 10A shows an example of a response from a large-scale language model provided in a RAG environment with knowledge information generated by the knowledge extraction process of the present invention, as a result of having a question answering system answer questions about the cause and countermeasures when damage occurs to the main machinery (main drive or spindle) of a hot rolling line in a steel mill.Figure 10B shows a comparison between the responses of a large-scale language model (not provided in the RAG environment) and a question-answering system, which was asked to provide answers regarding the causes and countermeasures in the case of damage to the main machinery (main drive and spindle) of a hot rolling line in a steel mill. Hereinafter, one embodiment of the present invention will be described with reference to the drawings. In all the drawings of the following embodiment, the same or corresponding parts are denoted by the same reference numerals. Furthermore, the present invention is not limited to the embodiment described below. (Embodiment) Figure 1 is a block diagram illustrating the schematic configuration of a knowledge processing system according to one embodiment of the present invention. As shown in Figure 1, the knowledge processing system 1 comprises a knowledge extraction device 10, a text generation device 20, and a server 30. The knowledge processing system 1 is configured such that the knowledge extraction device 10 and the text generation device 20 can input and output data from each other, and the knowledge extraction device 10 and the text generation device 20 can each read information from the server 30. In this embodiment, the knowledge extraction device 10 is configured with a question generation system and a knowledge extraction system using at least some of its components. Here, data input and output of the knowledge extraction device 10 and the text generation device 20 can be performed via network communication through a network or cloud, or via contactless communication such as Bluetooth®. It is also possible to move data via USB (Universal Serial Bus) memory, CD (Compact Disc), DVD (Digital Versatile Disc), or BD (Blu-ray® Disc) discs. The network is composed of a combination of wired and wireless communication as appropriate, and consists of communication networks such as the Internet network and mobile phone network. The network consists of one or more combinations of, for example, dedic