US-20260128040-A1 - SYSTEMS AND METHODS FOR AUTO-LABELING AND NATURAL LANGUAGE PROCESSING

Abstract

In some embodiments, the techniques described herein relate to a method including: receiving, by a text or call processing program, a plurality of seed utterances each with a plurality of predefined seed labels; receiving a first user utterance; converting the plurality of seed utterances and the first user utterance into n-gram phrases; grouping n-gram phrases of the plurality of seed utterances into phrase pools based on whether the n-gram phrases are 2-gram phrases or 3-gram phrases; executing n-gram matching of the user utterance with a first seed utterance of the plurality of seed utterances; and assigning a first intent label to the first user utterance based on a match of 3-gram phrases with the first seed utterance of the plurality of seed utterances.

Inventors

• Yungkwon KIM
• Di Wu
• Donald Stephens
• Lei Carol LIANG

Assignees

• JPMORGAN CHASE BANK, N.A.

Dates

Publication Date

20260507

Application Date

20241104

Claims (20)

1. 1 . A method performed by one or more computers, the method comprising: receiving, by a text or call processing program, a plurality of seed utterances each with a predefined seed label; receiving, by the text or call processing program, a first user utterance; converting, by the text or call processing program, the plurality of seed utterances and the first user utterance into n-gram phrases; grouping n-gram phrases of the plurality of seed utterances into phrase pools based on whether the n-gram phrases are 2-gram phrases or 3-gram phrases; executing, by the text or call processing program, n-gram matching of the first user utterance with a first seed utterance of the plurality of seed utterances; and assigning, by the text or call processing program, a first intent label to the first user utterance based on a match of 3-gram phrases with the first seed utterance of the plurality of seed utterances.
2. 2 . The method of claim 1 further comprising: filtering out, by the text or call processing program, non-frequent n-gram phrases in each phrase pool based on a predefined frequency threshold.
3. 3 . The method of claim 1 further comprising: receive, by the text or call processing program, a second user utterance; and marking, by the text or call processing program, a second user utterance with only one or more 2-gram matches for further processing.
4. 4 . The method of claim 3 further comprising: executing, by the text or call processing program, matching of the second user utterance with a second seed utterance of the plurality of seed utterances based on a pseudo-label of the second user utterance and the predefined seed label of the second seed utterance, the matching comprising: determining an output score comprising the 2-gram match and a distance between the second user utterance and the second seed utterance; and determining the output score exceeds a threshold score; and assigning, by the text or call processing program, the pseudo-label as a second intent label to the second user utterance based on the matching.
5. 5 . The method of claim 1 further comprising: receive, by the text or call processing program, a third user utterance; and marking, by the text or call processing program, the third user utterance with no n-gram matches for further processing.
6. 6 . The method of claim 5 further comprising: executing, by the text or call processing program, matching of the third user utterance with a third seed utterance of the plurality of seed utterances based on a pseudo-label of the third user utterance and the predefined seed label of the third seed utterance, the matching comprising: determining an output score comprising at least two distance measures between the third user utterance and the third seed utterance; and determining the output score exceeds a threshold score; and assigning, by the text or call processing program, the pseudo-label as a third label to the third user utterance based on the matching.
7. 7 . The method of claim 1 further comprising: determining, by the text or call processing program, a text response or an audible response based on a search of responses associated with the first intent label; and outputting, by the text or call processing program, the text response or the audible response.
8. 8 . A computer processing system comprising: a memory configured to store instructions; and a hardware processor operatively coupled to the memory for executing the instructions of a text or call processing program to: receive a plurality of seed utterances each with a predefined seed label; receive a first user utterance; convert the plurality of seed utterances and the first user utterance into n-gram phrases; group n-gram phrases of the plurality of seed utterances into phrase pools based on whether the n-gram phrases are 2-gram phrases or 3-gram phrases; execute n-gram matching of the first user utterance with a first seed utterance of the plurality of seed utterances; and assign a first intent label to the first user utterance based on a match of 3-gram phrases with the first seed utterance of the plurality of seed utterances.
9. 9 . The system of claim 8 , the instructions further comprising: filter out, by the text or call processing program, non-frequent n-gram phrases in each phrase pool based on a predefined frequency threshold.
10. 10 . The system of claim 8 , the instructions further comprising: receive a second user utterance; mark the second user utterance with only one or more 2-gram matches for further processing.
11. 11 . The system of claim 10 , the instructions further comprising: execute matching of the second user utterance with a second seed utterance of the plurality of seed utterances based on a pseudo-label of the second user utterance and the predefined seed label of the second seed utterance, the matching comprising: determining an output score comprising the 2-gram match and a distance between the second user utterance and the second seed utterance; and determining the output score exceeds a threshold score; and assign the pseudo-label as a second intent label to the second user utterance based on the matching.
12. 12 . The system of claim 8 , the instructions further comprising: receive a third user utterance; and mark the third user utterance with no n-gram matches for further processing.
13. 13 . The system of claim 12 , the instructions further comprising: execute matching of the third user utterance with a third seed utterance of the plurality of seed utterances based on a pseudo-label of the third user utterance and the predefined seed label of the third seed utterance, the matching comprising: determining an output score comprising at least two distance measures between the third user utterance and the third seed utterance; and determining the output score exceeds a threshold score; and assign, by the text or call processing program, the pseudo-label as a third label to the third user utterance based on the matching.
14. 14 . The system of claim 8 , the instructions further comprising: determining, by the text or call processing program, a text response or an audible response based on a search of responses associated with the first intent label; and outputting, by the text or call processing program, the text response or the audible response.
15. 15 . A non-transitory computer readable storage medium, including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising: receiving, by a text or call processing program, a plurality of seed utterances each with a predefined seed label; receiving, by the text or call processing program, a first user utterance; converting, by the text or call processing program, the plurality of seed utterances and the first user utterance into n-gram phrases; grouping n-gram phrases of the plurality of seed utterances into phrase pools based on whether the n-gram phrases are 2-gram phrases or 3-gram phrases; executing, by the text or call processing program, n-gram matching of the first user utterance with a first seed utterance of the plurality of seed utterances; and assigning, by the text or call processing program, a first intent label to the first user utterance based on a match of 3-gram phrases with the first seed utterance of the plurality of seed utterances.
16. 16 . The non-transitory computer readable storage medium of claim 15 further comprising: filtering out, by the text or call processing program, non-frequent n-gram phrases in each phrase pool based on a predefined frequency threshold.
17. 17 . The non-transitory computer readable storage medium of claim 15 further comprising: receiving a second user utterance; marking, by the text or call processing program, the second user utterance with only one or more 2-gram matches for further processing.
18. 18 . The non-transitory computer readable storage medium of claim 17 further comprising: executing, by the text or call processing program, matching of the second user utterance with a second seed utterance of the plurality of seed utterances based on a pseudo-label of the second user utterance and the predefined seed label of the second seed utterance, the matching comprising: determining an output score comprising the 2-gram match and a distance between the second user utterance and the second seed utterance; and determining the output score exceeds a threshold score; and assigning, by the text or call processing program, the pseudo-label as a second intent label to the second user utterance based on the matching.
19. 19 . The non-transitory computer readable storage medium of claim 15 further comprising: receiving, by the text or call processing program, a third user utterance; marking, by the text or call processing program, the third user utterance with no n-gram matches for further processing; executing, by the text or call processing program, matching of the third user utterance with a third seed utterance of the plurality of seed utterances based on a pseudo-label of the third user utterance and the predefined seed label of the third seed utterance, the matching comprising: determining an output score comprising at least two distance measures between the third user utterance and the third seed utterance; and determining the output score exceeds a threshold score; and assigning, by the text or call processing program, the pseudo-label as a third label to the third user utterance based on the matching.
20. 20 . The non-transitory computer readable storage medium of claim 15 further comprising: determining, by the text or call processing program, a text response or an audible response based on a search of responses associated with the first intent label; and outputting, by the text or call processing program, the text response or the audible response.

Description

BACKGROUND 1. Field of the Invention Embodiments generally relate to systems and methods for auto-labeling and natural language processing. 2. Description of the Related Art Companies receive thousands of calls from individuals seeking support, help, and access to a variety of company resources. The handling of those calls requires significant resources in human hours, bandwidth, and management hierarchy. Analysis of incoming calls and their contents can help a company to allocate resources according to customer needs. Appropriate analysis is theoretically based on accurate representation of what occurred during the call. Due to the nature of the analysis, companies develop models based on natural language processing to automated the process. However, to measure how accurate this automated system is not apparent since the data is human-languages, unstructured data that requires manual examination, which will be available on small samples only. Improved systems can drastically increase the coverage of data to be analyzed with much higher efficiency as well, leading to improve accuracy of tracking and analysis, and thus actions based on the tracking and analysis can be improved as a result. SUMMARY In some embodiments, the techniques described herein relate to receiving, by a text or call processing program, a plurality of seed utterances each with a predefined seed label; receiving, by the text or call processing program, a first user utterance; converting, by the text or call processing program, the plurality of seed utterances and the first user utterance into n-gram phrases; grouping n-gram phrases of the plurality of seed utterances into phrase pools based on whether the n-gram phrases are 2-gram phrases or 3-gram phrases; executing, by the text or call processing program, n-gram matching of the first user utterance with a first seed utterance of the plurality of seed utterances; and assigning, by the text or call processing program, a first intent label to the first user utterance based on a match of 3-gram phrases with the first seed utterance of the plurality of seed utterances. According to some embodiments, the method may further comprise filtering out, by the text or call processing program, non-frequent n-gram phrases in each phrase pool based on a predefined frequency threshold. According to some embodiments, the method may further comprise marking, by the text or call processing program, a second user utterance with only one or more 2-gram matches for further processing. According to some embodiments, the method may further comprise executing, by the text or call processing program, matching of the second user utterance with a second seed utterance of the plurality of seed utterances based on a pseudo-label of the second user utterance and the predefined seed label of the second seed utterance, the matching comprising: determining an output score comprising the 2-gram match and a distance between the second user utterance and the second seed utterance; and determining the output score exceeds a threshold score; and assigning, by the text or call processing program, the pseudo-label as a second intent label to the second user utterance based on the matching. According to some embodiments, the method may further comprise marking, by the text or call processing program, a third user utterance with no n-gram matches for further processing. According to some embodiments, the method may further comprise executing, by the text or call processing program, matching of the third user utterance with a third seed utterance of the plurality of seed utterances based on a pseudo-label of the third user utterance and the predefined seed label of the third seed utterance, the matching comprising: determining an output score comprising at least two distance measures between the third user utterance and the third seed utterance; and determining the output score exceeds a threshold score; and assigning, by the text or call processing program, the pseudo-label as a third label to the third user utterance based on the matching. According to some embodiments, the method may further comprise determining, by the text or call processing program, a text response or an audible response based on a search of responses associated with the first intent label; and outputting, by the text or call processing program, the text response or the audible response. Embodiments consistent with the present disclosure include a system including one or more processors and one or more storage devices storing instructions that when executed by one or more processors, cause the processor to perform one or more steps of the methods disclosed herein. Embodiments consistent with the present disclosure include a computer processing system, computer, or server, including: a memory configured to store instructions such as a non-transitory computer-readable storage medium; and a hardware processor operatively coupled to the memory for executing the