US-12626065-B2 - Lifecycle management for customized natural language processing

Abstract

Methods, systems, and computer-readable media for lifecycle management for customized natural language processing are disclosed. A natural language processing (NLP) customization service determines a task definition associated with an NLP model based (at least in part) on user input. The task definition comprises an indication of one or more tasks to be implemented using the NLP model and one or more requirements associated with use of the NLP model. The service determines the NLP model based (at least in part) on the task definition. The service trains the NLP model. The NLP model is used to perform inference for a plurality of input documents. The inference outputs a plurality of predictions based (at least in part) on the input documents. Inference data is collected based (at least in part) on the inference. The service generates a retrained NLP model based (at least in part) on the inference data.

Inventors

• Yahor Pushkin
• Rishita Rajal Anubhai
• Sameer Karnik
• Sunil Mallya Kasaragod
• Abhinav Goyal
• Yaser Al-Onaizan
• Ashish Singh
• Ashish Khare

Assignees

• AMAZON TECHNOLOGIES, INC.

Dates

Publication Date

20260512

Application Date

20200930

Claims (20)

1. 1 . A system, comprising: a natural language processing (NLP) customization service to select or build custom NLP models, the NLP customization service hosted by a provider network and Internet-accessible by a plurality of clients, wherein the provider network offers a plurality of services including the NLP customization service, and wherein the NLP customization service comprises one or more processors and one or more memories to store computer-executable instructions that, when executed, cause the one or more processors to: determine a task definition associated with an NLP model based at least in part on user input received via an interface, wherein the task definition comprises: an indication of one or more tasks to be implemented using the NLP model and one or more requirements for use of the NLP model in a production environment; select the NLP model, from a plurality of different candidate NLP model types having different levels of suitability for different task definitions, based at least in part on the task definition; train the NLP model using a plurality of training documents; evaluate, prior to deployment of the NLP model to the production environment, performance of the trained model against the one or more requirements indicated by the task definition; modify, based on the evaluation, the trained NLP model; deploy the modified NLP model to the production environment; perform inference using the NLP model for a plurality of input documents in the production environment, wherein the input documents comprise natural language text, wherein the inference outputs a plurality of predictions to a customer based at least in part on the natural language text, and wherein inference data is collected based at least in part on the inference performed in the production environment, wherein the inference data comprises: an indication of one or more input documents for which a prediction was generated with a confidence score below a confidence threshold, an indication of statistical deviation of one or more input documents with respect to the task definition from training documents used for training the model, or one or more performance metrics or resource usage metrics associated with the inference; and refine the NLP model based at least in part on automated analysis of accuracy or performance of the model, the accuracy or performance of the model based at least in part on the inference data.
2. 2 . The system as recited in claim 1 , wherein the task definition comprises a model accuracy threshold, and wherein the one or more memories store additional computer-executable instructions that, when executed, cause the one or more processors to: evaluate an accuracy of the NLP model with respect to the model accuracy threshold; and modify the NLP model based at least in part on evaluation of the accuracy of the NLP model.
3. 3 . The system as recited in claim 1 , wherein the task definition comprises one or more resource usage values, and wherein the one or more memories store additional computer-executable instructions that, when executed, cause the one or more processors to: select a model retraining rate based at least in part on the one or more resource usage values, wherein the NLP model is retrained according to the model retraining rate.
4. 4 . The system as recited in claim 1 , wherein the NLP model is configured to perform document classification or event extraction for the plurality of input documents.
5. 5 . A method, comprising: determining, by a natural language processing (NLP) customization service hosted by a provider network hosting a plurality of services and Internet-accessible by a plurality of clients, a task definition associated with an NLP model based at least in part on user input received via an interface, wherein the task definition comprises: an indication of one or more tasks to be implemented using the NLP model and one or more requirements for use of the NLP model; determining, by the NLP customization service and based at least in part on the task definition, the NLP model from a plurality of different candidate NLP model types having different levels of suitability for different task definition; training, by the NLP customization service, the NLP model; evaluating, prior to deployment of the NLP model to a production environment, performance of the trained model against one or more requirements indicated by the task definition; modifying, based on the evaluation, the trained NLP model; deploying the modified NLP model to the production environment; performing inference using the NLP model for a plurality of input documents, wherein the inference outputs a plurality of predictions based at least in part on the input documents, and wherein inference data is collected based at least in part on the inference performed in the production environment, wherein the inference data comprises: an indication of one or more input documents for which a prediction was generated with a confidence score below a confidence threshold, an indication of statistical deviation of one or more input documents with respect to the task definition from training documents used for training the model, or one or more performance metrics or resource usage metrics associated with the inference; and generating, by the NLP customization service and based at least in part on automated analysis of accuracy or performance of the model, a retrained NLP model, the accuracy or performance of the model based at least in part on the inference data.
6. 6 . The method as recited in claim 5 , wherein the task definition comprises a model accuracy threshold, and wherein the method further comprises: evaluating, by the NLP customization service, an accuracy of the NLP model with respect to the model accuracy threshold; and modifying, by the NLP customization service, one or more characteristics of the NLP model based at least in part on the evaluating.
7. 7 . The method as recited in claim 5 , wherein the task definition comprises one or more resource usage values, and wherein the method further comprises: selecting, by the NLP customization service, a model retraining rate based at least in part on the one or more resource usage values, wherein the NLP model is retrained according to the model retraining rate.
8. 8 . The method as recited in claim 5 , wherein the determining, by the NLP customization service, the NLP model based at least in part on the task definition further comprises: selecting, by the NLP customization service, a model topology from a plurality of candidate model topologies.
9. 9 . The method as recited in claim 5 , wherein the inference data comprises one or more input documents that statistically deviate with respect to the task definition from a corpus of training documents used in training the NLP model.
10. 10 . The method as recited in claim 5 , further comprising: collecting, by the NLP customization service, a plurality of training documents; annotating, by the NLP customization service, the plurality of training documents, wherein the NLP model is trained using the annotated training documents.
11. 11 . The method as recited in claim 5 , wherein the NLP model is deployed to a production environment after being trained, and wherein the method further comprises: deploying the retrained model to the production environment; and performing additional inference using the retrained NLP model in the production environment for a plurality of additional input documents, wherein additional inference data is collected based at least in part on the additional inference.
12. 12 . The method as recited in claim 5 , wherein the NLP customization service is hosted by a provider network and Internet-accessible by a plurality of clients via a service interface, and wherein the provider network offers a plurality of services including the NLP customization service.
13. 13 . One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors, perform: determining, by a natural language processing (NLP) customization service hosted by a provider network hosting a plurality of services and Internet-accessible by a plurality of clients, a task definition associated with an NLP model based at least in part on user input received via an interface, wherein the task definition comprises: an indication of one or more tasks to be implemented using the NLP model and one or more requirements for use of the NLP model in a production environment; selecting, by the NLP customization service, the NLP model from a plurality of different candidate NLP model types having different levels of suitability for different task definitions based at least in part on the task definition; training, by the NLP customization service, the NLP model; evaluating, prior to deployment of the NLP model to the production environment, performance of the trained model against one or more requirements indicated by the task definition; modifying, based on the evaluation, the trained NLP model; deploying the modified NLP model to the production environment; performing inference using the NLP model for a plurality of input documents, wherein the input documents comprise natural language text, wherein the inference outputs a plurality of predictions based at least in part on the natural language text, and wherein inference data is collected based at least in part on the inference, performed in the production environment, wherein the inference data comprises: an indication of one or more input documents for which a prediction was generated with a confidence score below a confidence threshold, an indication of statistical deviation of one or more input documents with respect to the task definition from training documents used for training the model, or one or more performance metrics or resource usage metrics associated with the inference; generating, by the NLP customization service, a retrained NLP model based at least in part on automated analysis of accuracy or performance of the model, the accuracy or performance of the model based at least in part on the inference data; and performing additional inference using the retrained NLP model for a plurality of additional input documents, wherein additional inference data is collected based at least in part on the additional inference.
14. 14 . The one or more non-transitory computer-readable storage media as recited in claim 13 , further comprising additional program instructions that, when executed on or across the one or more processors, perform: evaluating, by the NLP customization service, a performance of the NLP model with respect to the one or more requirements; and modifying, by the NLP customization service, one or more characteristics of the NLP model based at least in part on the evaluating.
15. 15 . The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein the task definition comprises one or more resource usage values, and wherein the one or more non-transitory computer-readable storage media further comprise additional program instructions that, when executed on or across the one or more processors, perform: selecting, by the NLP customization service, a training data processing rate based at least in part on the one or more resource usage values, wherein training data used in training the NLP model is collected, annotated, or corrected according to the training data processing rate.
16. 16 . The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein the inference data comprises one or more input documents that are associated with individual ones of the predictions having confidence scores below a confidence threshold.
17. 17 . The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein the NLP model is selected based at least in part on hyperparameter tuning.
18. 18 . The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein: said evaluating a performance of the NLP model and said deploying the NLP model to the production environment are performed an absence of additional user input from a user that specified the task definition.
19. 19 . The one or more non-transitory computer-readable storage media as recited in claim 13 , further comprising additional program instructions that, when executed on or across the one or more processors, perform: preparing, by the NLP customization service, a corpus of training documents, wherein the NLP model is trained using the corpus of training documents.
20. 20 . The one or more non-transitory computer-readable storage media as recited in claim 13 , wherein the NLP model is trained and evaluated based at least in part on a model building recipe, and wherein the model building recipe is selected based at least in part on the task definition.

Description

BACKGROUND Many companies and other organizations operate computer networks that interconnect numerous computing systems to support their operations, such as with the computing systems being co-located (e.g., as part of a local network) or instead located in multiple distinct geographical locations (e.g., connected via one or more private or public intermediate networks). For example, distributed systems housing significant numbers of interconnected computing systems have become commonplace. Such distributed systems may provide back-end services or systems that interact with clients. As the scale and scope of distributed systems have increased, the tasks of provisioning, administering, and managing system resources have become increasingly complicated. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A and FIG. 1B illustrate an example system environment for event extraction from documents with co-reference, according to some embodiments. FIG. 2 is an example of an input for event extraction from documents with co-reference, according to some embodiments. FIG. 3 is a flowchart illustrating a method for event extraction from documents with co-reference, according to some embodiments. FIG. 4A illustrates further aspects of the example system environment for event extraction from documents with co-reference, including an example of trigger detection, according to some embodiments. FIG. 4B illustrates further aspects of the example system environment for event extraction from documents with co-reference, including an example of entity detection, according to some embodiments. FIG. 4C illustrates further aspects of the example system environment for event extraction from documents with co-reference, including an example of role assignment, according to some embodiments. FIG. 5 illustrates further aspects of the example system environment for event extraction from documents with co-reference, including machine learning and annotation of documents, according to some embodiments. FIG. 6 illustrates an example system environment for context-sensitive linking of entities to private databases, according to some embodiments. FIG. 7 is an example of context-sensitive linking of entities to private databases, according to some embodiments. FIG. 8 illustrates further aspects of the example system environment for context-sensitive linking of entities to private databases, including the use of one or more extract-transform-load (ETL) tools, according to some embodiments. FIG. 9 illustrates further aspects of the example system environment for context-sensitive linking of entities to private databases, including the updating of entity representations and entity linking when corresponding records are changed in private databases, according to some embodiments. FIG. 10 illustrates further aspects of the example system environment for context-sensitive linking of entities to private databases, including the use of ranked contextual representations for filtering potential matches with records, according to some embodiments. FIG. 11 is a flowchart illustrating a method for context-sensitive linking of entities to private databases, according to some embodiments. FIG. 12A and FIG. 12B illustrate an example system environment for lifecycle management for customized natural language processing, according to some embodiments. FIG. 13 illustrates further aspects of the example system environment for lifecycle management for customized natural language processing, including tasks performed in model building and at runtime, according to some embodiments. FIG. 14A and FIG. 14B illustrate further aspects of the example system environment for lifecycle management for customized natural language processing, including a feedback loop for NLP model retraining, according to some embodiments. FIG. 15 illustrates further aspects of the example system environment for lifecycle management for customized natural language processing, including use of the NLP customization service to perform custom classification on input documents, according to some embodiments. FIG. 16 illustrates further aspects of the example system environment for lifecycle management for customized natural language processing, including use of the NLP customization service to perform custom extraction on input documents, according to some embodiments. FIG. 17 is a flowchart illustrating a method for lifecycle management for customized natural language processing, according to some embodiments. FIG. 18 illustrates an example computing device that may be used in some embodiments. While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cove