CN-121997939-A - Dialogue quality inspection method and equipment

Abstract

According to the dialogue quality inspection method and equipment provided by the embodiment of the specification, the conventional question-answering base of the first level and the professional knowledge base of the second level of the hierarchical knowledge base are preset, knowledge information corresponding to the user questions is searched from the hierarchical knowledge base according to the hierarchical order, and quality inspection is conducted on the question answers of the user questions based on the searched knowledge information. According to the technical scheme of the embodiment of the specification, the quality inspection of the dialogue data to be inspected of different types can be accurately performed by the layering quality inspection method, and the risk of missed inspection is remarkably reduced.

Inventors

• HU YUHAN

Assignees

• 支付宝(杭州)数字服务技术有限公司

Dates

Publication Date

20260508

Application Date

20260129

Claims (18)

1. 1. A conversation quality inspection method comprising: acquiring at least one to-be-checked-and-answered pair in a target scene, wherein the to-be-checked-and-answered pair comprises user questions in the target scene and question answers corresponding to the user questions; Retrieving knowledge information corresponding to the user questions of the question-answer pair to be checked from a hierarchical knowledge base in a hierarchical order, the hierarchical knowledge base comprising a first hierarchical regular question-answer base and a second hierarchical expert knowledge base, and And carrying out quality inspection on the question answers of the to-be-inspected question answer pair based on the retrieved knowledge information.
2. 2. The method of claim 1, wherein the retrieving knowledge information corresponding to the user question of the question-answer pair from a hierarchical knowledge base in a hierarchical order comprises: retrieving conventional questions corresponding to the user questions of the question-answer pair to be checked from the conventional question-answer library of the first level; And if the conventional problem is not retrieved, retrieving expertise corresponding to the user problem from the expertise base of the second hierarchy.
3. 3. The method of claim 2, wherein the retrieving, from the regular question-and-answer library of the first hierarchy, a regular question corresponding to the user question of the question-and-answer pair, comprises: Determining semantic similarity between the user questions of the question-answering pair and the conventional questions in the conventional question-answering library; If the semantic similarity corresponding to the conventional question is greater than or equal to a preset similarity threshold, acquiring a conventional question answer corresponding to the conventional question from the conventional question-answer library; And if the semantic similarity corresponding to each conventional question is smaller than the preset similarity threshold, determining that the conventional question is not retrieved from the conventional question-answering library.
4. 4. The method of claim 3, wherein the determining semantic similarity between the user question of the question-answer pair to be checked and the conventional question in the conventional question-answer library further comprises: Determining a preset weight of the conventional problem, the preset weight being a weight preset based on the importance of the conventional problem, and And carrying out weighted operation on the semantic similarity between the user problem and the conventional problem based on the preset weight.
5. 5. The method of claim 2, wherein the quality testing of the question answer of the question-answer pair based on the retrieved knowledge information comprises: And if the conventional questions are retrieved, performing quality inspection on the question answers corresponding to the user questions through preset quality inspection rules based on the conventional question answers corresponding to the conventional questions.
6. 6. A method according to claim 3, wherein said retrieving expertise corresponding to said user problem from said expertise base of said second hierarchy comprises: And retrieving the expertise corresponding to the user problem from the expertise base of the second hierarchy by multi-hop retrieval.
7. 7. The method of claim 6, wherein the retrieving expertise corresponding to the user question from the expertise base of the second hierarchy by multi-hop retrieval comprises: Decomposing the user question into a corresponding plurality of sub-questions; retrieving sub-specialized knowledge corresponding to a first one of the sub-questions from the specialized knowledge base based on the first one of the sub-questions; generating a current query corresponding to a current round based on the sub-questions of the current round and the sub-expertise of a previous round, and And searching sub-specialized knowledge corresponding to the current query from the specialized knowledge base based on the current query.
8. 8. The method of claim 1, wherein the quality testing of the question answer of the question-answer pair based on the retrieved knowledge information comprises: and at least inputting the retrieved knowledge information and the to-be-checked answer pair into a preset language model to guide the preset language model to check the quality of the question answer based on the knowledge information.
9. 9. The method of claim 8, wherein the inputting at least the retrieved knowledge information and the pair of questions to be asked into a pre-set language model comprises: and acquiring context information corresponding to the preset language model from a temporary storage area, and inputting at least the context information, the retrieved knowledge information and the to-be-checked-answer pair into the preset language model, wherein the temporary storage area is used for temporarily storing the context information in a vectorization mode.
10. 10. The method of claim 8, wherein the directing the pre-set language model to quality check the question answer based on the knowledge information comprises: generating a quality inspection prompt word by combining a quality inspection prompt word template based on the knowledge information and the to-be-inspected question-and-answer pair, wherein the quality inspection prompt word template is a preset prompt word template combining the knowledge information and the to-be-inspected question-and-answer pair to generate a quality inspection result, and And generating a quality inspection result corresponding to the question answer through the preset language model based on the quality inspection prompt word.
11. 11. The method of claim 8, wherein the predetermined language model is a language model obtained by knowledge distillation of a large language model.
12. 12. The method of claim 11, wherein the method further comprises: generating a plurality of question-answer pair samples in the target scene based on the large language model; Training the preset language model based on the plurality of question-answer pairs samples.
13. 13. The method of claim 11, wherein the output of the preset language model includes a confidence level for the quality test result of the challenge answer pair, the method further comprising: and if the confidence coefficient is smaller than a first confidence coefficient threshold value, triggering to manually review the to-be-inspected question-answer pair or to inspect the question answer of the to-be-inspected question-answer pair based on the large language model.
14. 14. The method of claim 13, wherein the method further comprises And generating a first correction sample based on the rechecking result of the manual rechecking so as to train the preset language model through the first correction sample.
15. 15. The method of claim 13, wherein after said quality testing of said question answer of said pair of questions to be tested based on said large language model, said method further comprises: If the confidence coefficient of the quality inspection result of the large language model for the question-answer pair sample is smaller than a second confidence coefficient threshold value, marking information of the question-answer pair sample is obtained, and a second correction sample is generated based on the marking information so as to train the preset language model through the second correction sample; And if the confidence coefficient of the quality inspection result of the question and answer to the sample by the large language model is larger than or equal to the second confidence coefficient threshold value, fine-tuning the preset language model according to the quality inspection result of the question and answer to the sample by the large language model.
16. 16. The method of claim 1, wherein the method further comprises: acquiring a history dialogue record in the target scene, wherein the history dialogue record comprises a plurality of question-answer pairs; And carrying out multidimensional clustering on the plurality of question-answer pairs to generate the conventional questions in the conventional question-answer library.
17. 17. The method of claim 1, wherein the method further comprises: and carrying out structured knowledge extraction on the domain expertise in the target scene, and generating the expertise in the expertise base.
18. 18. A dialog quality inspection device comprising: At least one storage medium storing at least one instruction set for performing a session quality check process, and At least one processor communicatively coupled to the at least one storage medium, Wherein the at least one processor reads the at least one instruction set and performs the dialog quality inspection method of any of claims 1-17 as directed by the at least one instruction set when the dialog quality inspection device is operating.

Description

Dialogue quality inspection method and equipment Technical Field The specification relates to the technical field of large models, in particular to a dialogue quality inspection method and dialogue quality inspection equipment. Background With the development of internet technology, the transaction amount and the client amount of a network platform are rapidly increasing, and the data amount of service dialogue data such as work order service data is increasing. How to perform quality detection on service session data on a network platform has become a focus of attention. In the related technical scheme, a plurality of basic service specification rules are preset, and whether the dialogue data to be detected accords with the service specification or not is checked according to the preset service specification rules through a regular expression. However, in this technical solution, only fixed keywords can be inspected by regular expressions, and it is difficult to accurately inspect the dialogue data to be inspected. Therefore, how to accurately perform quality inspection on dialogue data to be inspected becomes a technical problem to be solved. The statements in this background section merely provide information to the inventors and may not represent prior art to the present disclosure nor may they represent prior art to the filing date of the present disclosure. Disclosure of Invention The conversation quality inspection method and the conversation quality inspection device can accurately inspect the conversation data to be inspected, and reduce the risk of missed inspection. In a first aspect, the present specification provides a method for dialog quality inspection, including: acquiring at least one to-be-checked-and-answered pair in a target scene, wherein the to-be-checked-and-answered pair comprises user questions in the target scene and question answers corresponding to the user questions; Retrieving knowledge information corresponding to the user questions of the question-answer pair to be checked from a hierarchical knowledge base in a hierarchical order, the hierarchical knowledge base comprising a first hierarchical regular question-answer base and a second hierarchical expert knowledge base, and And carrying out quality inspection on the question answers of the to-be-inspected question answer pair based on the retrieved knowledge information. In some example embodiments, based on the above-mentioned scheme, the retrieving knowledge information corresponding to the user question of the answer pair to be checked from a hierarchical knowledge base according to a hierarchical order includes: retrieving conventional questions corresponding to the user questions of the question-answer pair to be checked from the conventional question-answer library of the first level; And if the conventional problem is not retrieved, retrieving expertise corresponding to the user problem from the expertise base of the second hierarchy. In some example embodiments, based on the above-described scheme, the retrieving, from the regular question-answer library of the first hierarchy, a regular question corresponding to the user question of the question-answer pair to be checked includes: Determining semantic similarity between the user questions of the question-answering pair and the conventional questions in the conventional question-answering library; If the semantic similarity corresponding to the conventional question is greater than or equal to a preset similarity threshold, acquiring a conventional question answer corresponding to the conventional question from the conventional question-answer library; And if the semantic similarity corresponding to each conventional question is smaller than the preset similarity threshold, determining that the conventional question is not retrieved from the conventional question-answering library. In some example embodiments, based on the above-described scheme, the determining the semantic similarity between the user question of the question-answer pair and the conventional question in the conventional question-answer library further includes: Determining a preset weight of the conventional problem, the preset weight being a weight preset based on the importance of the conventional problem, and And carrying out weighted operation on the semantic similarity between the user problem and the conventional problem based on the preset weight. In some example embodiments, based on the above-mentioned scheme, the quality inspection of the question answer of the question-answer pair based on the retrieved knowledge information includes: And if the conventional questions are retrieved, performing quality inspection on the question answers corresponding to the user questions through preset quality inspection rules based on the conventional question answers corresponding to the conventional questions. In some example embodiments, based on the above-described approach, the retrieving expertise