EP-4740150-A2 - SYSTEM AND METHOD OF AUTOMATICALLY GENERATING A NATURAL LANGUAGE WORKFLOW POLICY FOR A WORKFOW FOR CUSTOMER SUPPORT OF EMAILS

Abstract

A natural language workflow policy is generated for a workflow to solve customer support tickets is automatically generated from representative tickets. Tools, such as API calls, may also be automatically generated from representative tickets. In one implementation, a clustering technique may be used to identify representative answers for particular customer support topics. The generated workflows may be used by a large language model to generate answers for customer questions for an autonomous Al chatbot agent.

Inventors

• GHOCHE, SAMI
• Man, James
• Kong, Sunny
• Xing, Weitian
• Tosh, Zachary
• LU, ALAN
• Nicholas, Deon
• Hajal, Samy
• Chamoun, Jad
• Nasr, Antoine
• Lyubinets, Volodymyr
• Liao, EJ
• SHARMA, DEV
• LU, YI

Assignees

• Forethought Technologies, Inc.

Dates

Publication Date

20260513

Application Date

20240701

Claims (20)

1. 1. A method for generating a natural language workflow policy for responding to a customer service ticket comprising: identifying representative answers, in a collection of historic customer support tickets, for a question topic; inferring, using an Al model, a natural language workflow policy for solving customer support tickets for the question topic.
2. 2. The method of claim 1, wherein the identifying comprises clustering historic customer support tickets into topic clusters.
3. 3. The method of claim 2, further comprising forming answer clusters within a selected topic cluster.
4. 4. The method of claim 1, further comprising provided the natural language workflow policy for an administrator to perform at least one of selecting, customizing, and editing.
5. 5. The method of claim 1, further comprising: prompting a large language model with a natural language text description of the workflow policy, wherein the workflow policy is selected based on a topic of a customer support ticket to solve a specific customer issue; prompting the large language model with conversation information regarding a conversation between a customer an autonomous Al chatbot agent associated with the customer support ticket for the specific customer issue; prompting the large language model with information describing applicable actions to implement the workflow, including any applicable API calls; and prompting the large language model with at least one text message for the workflow; the large language model observing the results of actions of the workflow, making decisions on information to request from the customer, and making decisions on actions to take to implement the workflow.
6. 6. The method of claim 1, further comprising utilizing a taxonomy generated for historic customer support tickets to identify a workflow topic.
7. 7. The method of claim 1, wherein the natural language workflow further comprises at least one action having an associated software tool.
8. 8. The method of claim 7, wherein the associated software tool includes an API call.
9. 9. The apparatus of claim 1, wherein the natural language workflow includes at least one specified text message to respond to a particular instance of the workflow.
10. 10. The method of claim 1, wherein the autonomous Al chatbot agent interacts with the large language model.
11. 11. The method of claim 1, wherein autonomous Al chatbot agent includes the large language model.
12. 12. The method of claim 1, wherein the workflow policy comprises at least one natural language sentence.
13. 13. The method of claim 1, wherein the large language model is prompted with conversation information associated with a customer ticket, prompted with the workflow policy, prompted with information on applicable software tools for the workflow policy.
14. 14. The method of claim 13, wherein the large language model is further prompted with guard rail prompts.
15. 15. A method for responding to a customer service ticket comprising: analyzing historic customer support tickets and identifying representative answers for specific customer support topics; generating a natural language workflow policy for a particular customer support topic; determining a topic of a conversation for an autonomous Al chatbot agent responding to a customer service support ticket of a human being corresponds to the particular topic; utilizing a large language model to implement a workflow to solve the customer service support ticket; and prompting the large language model with the workflow policy and a description of available tools to generate an interactive workflow for interacting with the customer to solve the customer service support ticket.
16. 16. The method of claim 15, further comprising utilizing a taxonomy generated for historic customer support tickets to generate a list of customer support tickets.
17. 17. The method of claim 15, wherein the natural language workflow further comprises at least one action having an associated software tool.
18. 18. The method of claim 16, wherein the associated software tool includes an API call.
19. 19. The apparatus of claim 15, wherein the natural language workflow includes at least one specified text message to respond to a particular instance of the workflow.
20. 20. The method of claim 15, wherein the autonomous Al chatbot agent interacts with the large language model.

Description

SYSTEM AND METHOD OF AUTOMATICALLY GENERATING A NATURAL LANGUAGE WORKFLOW POLICY FOR A WORKFOW FOR CUSTOMER SUPPORT OF EMAILS FIELD OF THE INVENTION [0001] The present disclosure generally relates to servicing customer support issues, such as responding to questions or complaints during a lifecycle of a customer support issue. BACKGROUND [0002] Customer support service is an important aspect of many businesses. For example, there are a variety of customer support applications to address customer service support issues. As one illustration, a customer service helpdesk may have a set of human agents who use text messages to service customer support issues. There are a variety of Customer Relationship Management (CRM) and helpdesk-related software tools, such as SalesForce® or Zendesk®. [0003] Customer support issues may be assigned a ticket that is served by available human agents over the lifecycle of the ticket. The lifecycle of resolving the customer support issue(s) associated with a ticket may include one or more customer questions and one or more answers made by an agent in response to customer question(s). To address common support questions, the human agents may have available to them macros and templates in SalesForce® or templates in Zendesk® as examples. Macros and templates work well for generating information to respond to routine requests for information, such as if a customer asks, “Do you offer refunds?” However, there are some types of more complicated or nonroutine questions for which there may be no macro or template. [0004] Human agents may have available to them other data sources spread across an organization (e.g., Confluence®, WordPress®, Nanorep®, Readmeio®, JIRA®, Guru®, Knowledge Bases, etc.). However, while an institution may have a lot of institutional knowledge to aid human agents, there may be practical difficulties in training agents to use all the institutional knowledge that is potentially available to aid in responding to tickets. For example, conventionally, a human agent may end up doing a manual search of the institutional knowledge. However, an agent may waste time in unproductive searches of the institutional knowledge. [0005] Typically, a human expert makes decisions on how to label and route tickets, which is a resource intensive task. There is also a delay associated with this process because incoming tickets have to wait in a queue for a human expert to make labeling and routing decisions. [0006] However, there are substantial training and labor costs to have a large pool of highly trained human agents available to service customer issues. There are also labor costs associated with having human experts making decisions about how to label and route tickets. But in addition to labor costs, there are other issues in terms of the frustration customers experience if there is a long delay in responding to their queries. [0007] In addition to other issues, it has often been impractical in conventional techniques to have more than a small number of customer issue topics as categories. That is. conventionally tickets are categorized into a small number of categories (e.g., 15) for handling by agents. BRIEF DESCRIPTION OF THE DRAWINGS [0008] The present disclosure is illustrated by way of example, and not by way of limitation in the figures of the accompanying drawings in which like reference numerals are used to refer to similar elements. [0009] Fig. 1 is a block diagram illustrating a customer service support environment in accordance with an implementation. [0010] Fig. 2A is a block diagram illustrating a module for using Al to augment customer support agents in accordance with an implementation. [0011] Fig. 2B is a block diagram of a server-based implementation. [0012] Fig. 3 is a block diagram of a portion of a ML system in accordance with an implementation. [0013] Fig. 4 is a flow chart of a method of servicing tickets in accordance with an implementation. [0014] Fig. 5 is a flow chart of a method of automatically generating a templet answer to an incoming question in accordance with an implementation. [0015] Fig. 6 illustrates an example of ML pipeline in accordance with an implementation. [0016] Fig. 7 illustrates aspects of using supervised learning to solve macros in accordance with an implementation. [0017] Fig. 8 is a flow chart of a method of generating macro template answer codes in accordance with an implementation. [0018] Fig. 9 illustrates an example of a method of identifying macro template answers and also initiating a workflow task in accordance with an implementation. [0019] Fig. 10 illustrates an example of a method of training a ML model for triaging the routing of tickets in accordance with an implementation. [0020] Fig. 11 illustrates a method of performing triaging in the routing of tickets in accordance with an implementation. [0021] Fig. 12 illustrates a method of identifying knowledge based articles to respond to a question in accordance with