CN-122019886-A - Quality evaluation method and device for search recommended service

Abstract

One or more embodiments of the present disclosure provide a method and an apparatus for evaluating quality of a search recommended service. According to the method, search recommendation service data to be evaluated can be obtained, the search recommendation service data comprises request information and corresponding M search recommendation results, then correlation evaluation is carried out on the request information with the first N results and the M results in the search recommendation results through an evaluation model, first correlation results corresponding to the first N results and second correlation results corresponding to the M results are obtained, error correction evaluation is carried out on the search recommendation service based on the request information through the evaluation model, error correction evaluation results are obtained, and finally quality evaluation results of the search recommendation service are obtained based on the first correlation results, the second correlation results and the error correction evaluation results.

Inventors

• CAI LISHAN
• BU YIFAN
• LI XIAOHONG

Assignees

• 支付宝(杭州)数字服务技术有限公司

Dates

Publication Date

20260512

Application Date

20260213

Claims (20)

1. 1. A quality evaluation method of a search recommended service, the method comprising: The method comprises the steps of obtaining search recommendation service data to be evaluated, wherein the search recommendation service data to be evaluated comprises request information and search recommendation results, the search recommendation results are a set of M results output by a search recommendation service based on the request information, and M is a positive integer; Performing correlation evaluation on the request information with the first N results and the M results in the search recommendation results through an evaluation model to obtain a first correlation result corresponding to the first N results and a second correlation result corresponding to the M results, wherein N is a positive integer less than or equal to M; performing error correction evaluation on the search recommended service based on the request information through the evaluation model to obtain an error correction evaluation result; And obtaining a quality evaluation result of the search recommended service based on the first correlation result, the second correlation result and the error correction evaluation result.
2. 2. The method for evaluating the quality of a search recommended service according to claim 1, wherein the quality evaluation result includes an evaluation score, the method further comprising: Marking the evaluation result with the evaluation score lower than a score threshold as an abnormal result; And carrying out attribution diagnosis on the abnormal result to obtain an attribution result.
3. 3. The method for evaluating the quality of a search recommended service according to claim 2, wherein said performing attribution diagnosis on the abnormal result to obtain an attribution result comprises: diagnosing the abnormal result based on a preset attribution rule to obtain a first attribution result, wherein the first attribution result comprises an abnormal reason and a confidence coefficient; And diagnosing the abnormal result based on a diagnosis model under the condition that the confidence degree of the first attribution result is lower than a confidence degree threshold value, so as to obtain a second attribution result.
4. 4. The method for evaluating the quality of a search recommended service according to claim 3, wherein performing correlation evaluation on the request information with a first N results and the M results in the search recommended results, respectively, to obtain a first correlation result corresponding to the first N results and a second correlation result corresponding to the M results, includes: Performing relevance matching on the request information and each result in the search recommendation results to obtain relevance scores of each result; And obtaining the first correlation scores corresponding to the first N results and the second correlation scores corresponding to the M results based on the correlation scores of the results.
5. 5. The method for evaluating the quality of a search recommendation service according to claim 4, wherein the request information includes current location information and target location information, and wherein the performing relevance matching on the request information and each result in the search recommendation results to obtain a relevance score of each result includes: analyzing the request information to obtain a request intention; Based on the request intention, carrying out geographic relevance matching on the current position information or the target position information and the service coverage corresponding to each result to obtain a geographic relevance score of each result; Carrying out semantic relevance matching on the request intention and the service description information corresponding to each result to obtain a semantic relevance score of each result; And weighting the geographic relevance score and the semantic relevance score to obtain the relevance score of each result.
6. 6. The method for evaluating the quality of a search recommended service according to claim 5, wherein the request information includes query information input by a user or recommendation information of the search recommended service, and the performing error correction evaluation on the search recommended service based on the request information to obtain an error correction evaluation result includes: rewriting the query information input by the user to generate request sample information containing preset deviation; Acquiring error correction information obtained by performing error correction processing on the request sample information by the search recommendation service; And carrying out semantic retention comparison on the request sample information and the error-corrected information, and obtaining an error-correction evaluation score based on a comparison result.
7. 7. The method for evaluating the quality of a search recommended service according to claim 6, wherein the obtaining the quality evaluation result of the search recommended service based on the first correlation result, the second correlation result, and the error correction evaluation result comprises: Weighting the first correlation result and the second correlation result to obtain a correlation evaluation result; obtaining a model evaluation result based on the correlation evaluation result and the error correction evaluation result; acquiring a preset evaluation rule, and performing multidimensional evaluation on the search recommendation result based on the preset evaluation rule to obtain a rule evaluation result; and obtaining the quality evaluation result of the search recommended service based on the model evaluation result and the rule evaluation result.
8. 8. The method for evaluating the quality of the search recommended service according to claim 7, wherein the preset evaluation rules comprise an availability evaluation rule, an integrity evaluation rule and a diversity evaluation rule, and the multi-dimensional evaluation is performed on the search recommended result based on the preset evaluation rules to obtain a rule evaluation result, wherein the rule evaluation result comprises at least two of the following: based on the usability evaluation rule, usability evaluation is carried out on the search recommendation result, and a usability evaluation result is obtained; Based on the integrity evaluation rule, carrying out integrity evaluation on the search recommendation result to obtain an integrity evaluation result; and carrying out diversity evaluation on the search recommendation result based on the diversity evaluation rule to obtain a diversity evaluation result.
9. 9. The method for evaluating the quality of a search recommendation service according to claim 8, wherein the search recommendation service data to be evaluated further comprises a recall result, the recall result is a candidate result set recalled by the search recommendation service based on the request information, the search recommendation result is obtained by filtering based on the recall result, the usability evaluation rule based on the usability evaluation rule is used for evaluating the usability of the search recommendation result, and the method comprises the following steps: acquiring a first number of recall results and a second number of search recommendation results; Determining that the availability evaluation result is a recall-free result under the condition that the first number is zero; Determining that the usability evaluation result is a non-output result under the condition that the first quantity is not zero and the second quantity is zero; and under the condition that the request information is not matched with the service scene of the search recommendation service, determining that the usability evaluation result is an unexpected result.
10. 10. The method for evaluating the quality of the search recommended service according to claim 8, wherein the performing integrity evaluation on the search recommended result based on the integrity evaluation rule to obtain an integrity evaluation result comprises: Information filtering processing is carried out on the search recommendation results to obtain the number of filtered results; And under the condition that the number of the filtered results is smaller than a first number threshold, determining that the integrity evaluation result is insufficient.
11. 11. The method for evaluating the quality of a search recommended service according to claim 8, wherein the performing diversity evaluation on the search recommended result based on the diversity evaluation rule to obtain a diversity evaluation result includes: Determining whether the request information contains a service category; Acquiring the number of service categories contained in the search recommendation result when the request information does not contain the service categories and the ratio of any one of the service categories in the search recommendation result exceeds a first ratio threshold; and if the number of the service categories is smaller than a second number threshold, determining that the diversity evaluation result is insufficient in diversity.
12. 12. The method for evaluating the quality of a search recommended service according to claim 9, wherein the preset attribution rules include a correlation attribution rule, a no-result attribution rule, and a geographic attribution rule, wherein diagnosing the abnormal result based on the preset attribution rule comprises: Diagnosing the abnormal result based on the relevance attribution rule in the case that the ratio of the result with the relevance score lower than a first score threshold value in the search recommendation result exceeds a second proportion threshold value; The availability evaluation result is the search recommended service data without recall result, the output result or the expected result, and diagnosis is carried out on the abnormal result based on the no-result attribution rule; And diagnosing the abnormal result based on the geographic attribution rule for the search recommended service data with the geographic relevance score lower than a second score threshold.
13. 13. The method for evaluating the quality of a search recommended service according to claim 12, wherein the diagnosing the abnormal result based on the correlation attribution rule comprises: Acquiring a first result number of which the first relevance score exceeds a third score threshold and a second result number of which the second relevance score exceeds the third score threshold in the search recommendation results; And obtaining the first attribution result based on the first result number and the second result number.
14. 14. The method for evaluating the quality of a search recommended service according to claim 12, wherein the diagnosing the abnormal result based on the no-result attribution rule comprises: verifying the request intention obtained by analyzing the request information, and determining whether the request intention is analyzed correctly; And diagnosing the search recommendation result based on the request sample information corresponding to the request information to obtain the first attribution result under the condition that the request intention analysis is correct.
15. 15. The method for evaluating the quality of a search recommended service according to claim 12, wherein the diagnosing the abnormal result based on the geo-attribution rule comprises: acquiring a target city code obtained based on the analysis of the request information, and performing consistency check on the target city code and the current position information or the target position information; Analyzing the interest points of the target position information to obtain interest point identifiers, and checking the accuracy of the interest point identifiers and preset interest point identifiers; and obtaining the first attribution result based on the consistency check result and the accuracy check result.
16. 16. The method for evaluating the quality of a search recommended service according to claim 1, characterized in that the method further comprises: After each search recommendation service is completed, asynchronously acquiring identification information of the search recommendation service based on a message queue; And acquiring service data corresponding to the search recommended service in the log file based on the identification information.
17. 17. The method for evaluating the quality of a search recommended service according to claim 2, characterized in that the method further comprises: based on the quality evaluation result, generating a visual chart corresponding to the search recommendation service; And/or the number of the groups of groups, And sending out an abnormal alarm under the condition that the quality evaluation result or the attribution result meets a preset abnormal condition.
18. 18. A quality evaluation apparatus for searching for recommended services, the apparatus comprising: The acquisition module is used for acquiring search recommendation service data to be evaluated, wherein the search recommendation service data to be evaluated comprises request information and search recommendation results, the search recommendation results are a set of M results output by the search recommendation service based on the request information, and M is a positive integer; The correlation evaluation module is used for performing correlation evaluation on the request information, the first N results and the M results in the search recommendation results through an evaluation model to obtain a first correlation result corresponding to the first N results and a second correlation result corresponding to the M results, wherein N is a positive integer smaller than or equal to M; The error correction evaluation module is used for performing error correction evaluation on the search recommended service based on the request information through the evaluation model to obtain an error correction evaluation result; And the result integration module is used for obtaining the quality evaluation result of the search recommended service based on the first correlation result, the second correlation result and the error correction evaluation result.
19. 19. An electronic device, the electronic device comprising: a memory for storing a computer program product; a processor for executing the computer program product stored in the memory, and when executed, implementing the quality assessment method of a search recommended service according to any of the preceding claims 1 to 17.
20. 20. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon computer program instructions which, when executed, implement the quality evaluation method of a search recommended service according to any one of the preceding claims 1 to 17.

Description

Quality evaluation method and device for search recommended service Technical Field One or more embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method and an apparatus for evaluating quality of a search recommended service. Background With the rapid development of information technology, digital service products enrich the life scenes of people. The searching recommendation system becomes a core carrier for connecting users and services and is widely applied to multiple scenes such as electronic commerce, information, life services and the like. In order to adapt to the characteristics of high request quantity and quick iteration of a search recommendation system, meet the requirements of real-time perception and comprehensive evaluation of the search recommendation service quality and accurate positioning of the problem root, a search recommendation service quality evaluation scheme which is efficient and fits an actual scene needs to be provided, so that timeliness and pertinence of quality evaluation are improved, and a powerful support is provided for optimization of the system. Disclosure of Invention In order to meet the requirements of real-time sensing, comprehensive evaluation and accurate positioning of the root cause of the problems of the search recommended service quality, one or more embodiments of the present specification provide a quality evaluation method and apparatus for the search recommended service. In a first aspect, one or more embodiments of the present disclosure provide a quality evaluation method for searching for recommended services, the method including: The method comprises the steps of obtaining search recommendation service data to be evaluated, wherein the search recommendation service data to be evaluated comprises request information and search recommendation results, the search recommendation results are a set of M results output by a search recommendation service based on the request information, and M is a positive integer; Performing correlation evaluation on the request information with the first N results and the M results in the search recommendation results through an evaluation model to obtain a first correlation result corresponding to the first N results and a second correlation result corresponding to the M results, wherein N is a positive integer less than or equal to M; performing error correction evaluation on the search recommended service based on the request information through the evaluation model to obtain an error correction evaluation result; And obtaining a quality evaluation result of the search recommended service based on the first correlation result, the second correlation result and the error correction evaluation result. In one possible implementation, the quality evaluation result includes an evaluation score, and the method further includes: Marking the evaluation result with the evaluation score lower than a score threshold as an abnormal result; And carrying out attribution diagnosis on the abnormal result to obtain an attribution result. In one possible implementation manner, the performing attribution diagnosis on the abnormal result to obtain an attribution result includes: diagnosing the abnormal result based on a preset attribution rule to obtain a first attribution result, wherein the first attribution result comprises an abnormal reason and a confidence coefficient; And diagnosing the abnormal result based on a diagnosis model under the condition that the confidence degree of the first attribution result is lower than a confidence degree threshold value, so as to obtain a second attribution result. In a possible implementation manner, the performing relevance evaluation on the request information with the first N results and the M results in the search recommendation results to obtain a first relevance result corresponding to the first N results and a second relevance result corresponding to the M results includes: Performing relevance matching on the request information and each result in the search recommendation results to obtain relevance scores of each result; And obtaining the first correlation scores corresponding to the first N results and the second correlation scores corresponding to the M results based on the correlation scores of the results. In one possible implementation manner, the request information includes current location information and target location information, and the performing relevance matching on the request information and each result in the search recommendation results to obtain a relevance score of each result includes: analyzing the request information to obtain a request intention; Based on the request intention, carrying out geographic relevance matching on the current position information or the target position information and the service coverage corresponding to each result to obtain a geographic relevance score of each result; Carr