US-20260127462-A1 - OBTAINING AND UTILIZING FEEDBACK FOR AGENT-ASSIST SYSTEMS

US20260127462A1US 20260127462 A1US20260127462 A1US 20260127462A1US-20260127462-A1

Abstract

Techniques for agent-assist systems to provide context-aware, subdocument-granularity recommended answers to agents that are attempting to answer queries of users. The agent-assist system may obtain collections of documents that include information for responding to queries, and analyze those documents to identify subdocuments that are associated with different semantics or meanings. Subsequently, any queries received can be analyzed to identify their semantics, and relevant subdocuments can be identified as having similar semantics. When the agent-assist system presents the agent with the relevant documents, it may highlight or otherwise indicate the relevant subdocument within the document for quick identification by the agent. Further, the agent-assist system may collect feedback from the agent and/or user to determine a relevancy of the recommended answers. The agent-assist system can use the feedback to improve the quality of the recommended answers provided to the agents.

Inventors

Mohamed Gamal Mohamed Mahmoud
Elizabeth Hutton
Bhavana Bhasker
Muthu Kumaran Ponnambalam
Puneet Shrivastava
Duraikrishna Selvaraju

Assignees

CISCO TECHNOLOGY, INC.

Dates

Publication Date: 20260507
Application Date: 20251231

Claims (20)

1 . A method performed at least partly by an agent system of a contact center, the method comprising: obtaining documents at the agent system of the contact center; converting the documents into mathematical representations of semantic meanings of the documents; storing the mathematical representations of the semantic meanings of the documents in a knowledge base of the agent system; establishing a communication session between a user device and an agent device, wherein the agent device runs software that supports the agent system; receiving, from the user device, first input from the user engaged in the communication session; identifying, from the first input, a query that the user has for an agent to answer; retrieving, using a vector-based semantic search algorithm, context data from the knowledge base of the agent system that is relevant to the query, wherein the retrieving the context data comprises: generating a first vector representation for the query; determining a distance between the first vector representation and a second vector representation stored in the knowledge base, the second vector representation corresponding to the context data; and selecting the context data based on the distance; providing the context data as second input into a generative model; receiving, as output from the generative model, an answer for the query; and providing the answer to the user device via the communication session.
2 . The method of claim 1 , wherein the converting the documents into mathematical representations of semantic meanings comprises: identifying subdocuments from each of the documents, wherein each subdocument includes a portion of text from a respective document that is less than all of the text in the respective document; and generating an embedding for each subdocument, wherein each embedding represents a semantic meaning of a corresponding subdocument.
3 . The method of claim 2 , wherein the identifying subdocuments comprises: determining topic drift within each document by comparing embeddings of sequential text units; and establishing subdocument boundaries based on the topic drift exceeding a similarity threshold.
4 . The method of claim 1 , wherein the distance is a cosine similarity distance between the first vector representation and the second vector representation in a high-dimensional vector space.
5 . The method of claim 1 , further comprising: filtering the query prior to the retrieving, wherein the filtering comprises determining whether the query is relevant to a domain associated with the knowledge base.
6 . The method of claim 5 , wherein the filtering comprises: detecting whether the query is small talk using a binary classifier; and in response to detecting that the query is small talk, preventing the query from being processed by the vector-based semantic search algorithm.
7 . The method of claim 1 , wherein the retrieving the context data further comprises: generating a conversation-context-aware query embedding by combining the first vector representation with prior query embeddings from the communication session.
8 . The method of claim 1 , further comprising: receiving feedback indicating a relevancy of the answer for responding to the query; and adjusting a confidence value associated with the context data based on the feedback, wherein the confidence value indicates a likelihood that the context data is relevant for responding to similar queries.
9 . The method of claim 8 , wherein the feedback comprises implicit feedback determined by comparing text of the answer provided by the generative model with text of a response provided by an agent associated with the agent device.
10 . A system comprising: one or more processors; and one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: obtaining documents at an agent system of a contact center; converting the documents into mathematical representations of semantic meanings of the documents; storing the mathematical representations of the semantic meanings of the documents in a knowledge base of the agent system; establishing a communication session between a user device and an agent device, wherein the agent device runs software that supports the agent system; receiving, from the user device, first input from a user engaged in the communication session; identifying, from the first input, a query that the user has for an agent to answer; retrieving, using a vector-based semantic search algorithm, context data from the knowledge base of the agent system that is relevant to the query, wherein the retrieving the context data comprises: generating a first vector representation for the query; determining a distance between the first vector representation and a second vector representation stored in the knowledge base, the second vector representation corresponding to the context data; and selecting the context data based on the distance; providing the context data as second input into a generative model; receiving, as output from the generative model, an answer for the query; and providing the answer to the user device via the communication session.
11 . The system of claim 10 , wherein the converting the documents into mathematical representations of semantic meanings comprises: identifying subdocuments from each of the documents, wherein each subdocument includes a portion of text from a respective document that is less than all of the text in the respective document; and generating an embedding for each subdocument, wherein each embedding represents a semantic meaning of a corresponding subdocument.
12 . The system of claim 11 , wherein the identifying subdocuments comprises: determining topic drift within each document by comparing embeddings of sequential text units; and establishing subdocument boundaries based on the topic drift exceeding a similarity threshold.
13 . The system of claim 10 , wherein the distance is a cosine similarity distance between the first vector representation and the second vector representation in a high-dimensional vector space.
14 . The system of claim 10 , the operations further comprising: filtering the query prior to the retrieving, wherein the filtering comprises determining whether the query is relevant to a domain associated with the knowledge base.
15 . The system of claim 14 , wherein the filtering comprises: detecting whether the query is small talk using a binary classifier; and in response to detecting that the query is small talk, preventing the query from being processed by the vector-based semantic search algorithm.
16 . The system of claim 10 , the operations further comprising: receiving feedback indicating a relevancy of the answer for responding to the query; and adjusting a confidence value associated with the context data based on the feedback, wherein the confidence value indicates a likelihood that the context data is relevant for responding to similar queries.
17 . A method performed at least partly by an agent system, the method comprising: obtaining documents at the agent system; identifying subdocuments from the documents, wherein each subdocument comprises a portion of text from a respective document that is less than all text in the respective document; generating embedding vectors for the subdocuments, wherein each embedding vector represents a semantic meaning of a corresponding subdocument; storing the embedding vectors in a vector space of a knowledge base; receiving input representing a query from a user; generating a query embedding vector representing a semantic meaning of the query; identifying, from the vector space, a target embedding vector that is within a threshold distance of the query embedding vector; retrieving a target subdocument corresponding to the target embedding vector; providing the target subdocument as context to a generative model along with the query; receiving, from the generative model, a generated response based on the context and the query; and outputting the generated response.
18 . The method of claim 17 , wherein the identifying subdocuments comprises: determining topic drift within each document by comparing embedding vectors of sequential text units; and establishing subdocument boundaries based on the topic drift exceeding a similarity threshold.
19 . The method of claim 17 , further comprising: filtering the query prior to the identifying the target embedding vector, wherein the filtering comprises determining whether the query is relevant to a domain associated with the knowledge base.
20 . The method of claim 19 , wherein the filtering comprises: detecting whether the query is small talk using a binary classifier; and in response to detecting that the query is small talk, preventing the query from being processed.

Description

RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. application Ser. No. 17/332,099, filed on May 27, 2021, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD The present disclosure relates generally to an agent-assist system that provides context-aware recommendations to agents that are attempting to answer queries of users, and improves the quality of the recommended answers provided to agents when responding to the queries of users. BACKGROUND Across many industries, contact centers are used for receiving large volumes of inquiries from users (often customers) of different products, services, and various offerings from companies or organizations. Contact centers can be centralized offices, and/or decentralized offices (e.g., remote agents), used for receiving inbound communications from users (e.g., telephone calls, Short Message Service (SMS) messages, etc.), and dispatching those inbound communications to agents that are trained to help users with their inquiries. However, these contact centers often field inquires across many different topics or domains, such as for different companies, different products, different industries, and so forth. When it comes to solving user issues and answering user inquiries, a key skill for contact-center agents is domain and technical expertise. To obtain domain and technical expertise, these agents often need extensive training and experience to build expertise in a specific domain. However, contact centers often face a lot of turnover or churn in their agents, and it can be difficult to find, train, and keep agents that are skilled in specific domains. As a result, agents handing user-support calls typically need assistance in the domain. Typically, agents find responses to queries by looking up the answers in documents, searching knowledge bases, and consulting with other agents in the contact center. However, this increases call handling time, decreases agent productively, and can frustrate customers or users. To help alleviate the burden on agents, agent-assist systems have been developed to help agents that are handling user interactions by recommending resources or answers that are relevant to a user's issue or inquiry. The purpose of these agent-assist systems is to decrease the average call handling time (AHT), increase the resolution rate for user interactions, minimize agent training time, and provide fast and accurate sources of information to agents. However, agent-assist systems can provide recommended answers or documents that are not pertinent to the query, which decreases performance by the agent and reduces user satisfaction. Additionally, the agent-assist systems can surface information or answers that are not relevant to the inquires received from the user. Accordingly, agent-assist systems may not actually provide helpful or useful assistance to agents in some scenarios. BRIEF DESCRIPTION OF THE DRAWINGS The detailed description is set forth below with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items. The systems depicted in the accompanying figures are not to scale and components within the figures may be depicted not to scale with each other. FIG. 1 illustrates a system-architecture diagram of an example environment in which an agent-assist system provides recommended answers for an agent of a contact-center environment to use when responding to a query from a user. FIG. 2 illustrates an example system-architecture diagram of an agent-assist pipeline that provides recommended answers to an agent device for an agent to use when responding to a query of a user. FIG. 3 illustrates an example flow diagram according to which an agent-assist pipeline extracts subdocuments from a plurality of documents, represents the subdocuments as semantic vectors, and creates a vector space with the semantic vectors. FIG. 4A illustrates a graphical user interface through which an agent is presented with recommended answers from an agent-assist system, and is able to identify relevant answers at subdocument granularity for responding to a query of a user. FIG. 4B illustrates a graphical user interface through which an agent views recommended answers provided by an agent-assist system, and provides explicit and/or implicit feedback regarding the relevancy of the recommended answers for responding to a query of a user. FIG. 5 illustrates an example system-architecture diagram of a feedback pipeline that receives feedback provided by an agent indicating a relevancy of recommended answers for responding to a query, and modifies rankings of the recommended answers based on the feedback. FIG. 6 illustrates a flow diagram of an example method for an agent-assist system to obtain collections of doc