Search

US-12619399-B1 - Requirements discovery for generative ai software development assistant

US12619399B1US 12619399 B1US12619399 B1US 12619399B1US-12619399-B1

Abstract

Techniques for leveraging a large language model (LLM) in software development are described. A description of a software development task is received from a user. Data associated with the software system is obtained from a data source. An LLM is prompted to identify at least one aspect of the task which requires clarification from the user, at least partly by providing the obtained data to the LLM and asking the LLM to identify a question for the user which remains unanswered by the obtained data. The question is presented to the user. An answer to the question is received from the user. The LLM is prompted to respond to propose an implementation of the task at least partly based on the data associated with the software system and the answer received from the user. The proposed implementation is received from the LLM and caused to be displayed to the user.

Inventors

  • Mark Rambow
  • Jonathan Weiss

Assignees

  • AMAZON TECHNOLOGIES, INC.

Dates

Publication Date
20260505
Application Date
20230630

Claims (20)

  1. 1 . A system comprising: a large language model (LLM) hosted in a multi-tenant provider network; and a software development service (SDS) provided by the multi-tenant provider network, the SDS executing SDS code using one or more processors to cause the SDS to: receive, by a system design agent of the executing SDS code from a user, a description of a software development task to make a change to a software system; prompt, by the system design agent of the executing SDS code, the LLM with a first prompt to generate a set of questions related to the software development task, wherein the first prompt includes the description of the software development task; receive, by the system design agent of the executing SDS code, the set of questions from the LLM; obtain, by one or more data-gathering context aggregators of the executing SDS code in response to receiving the description of the software development task to make the change to the software system, data associated with at least the software system by one or more of (i) accessing source code of the software system, (ii) accessing documentation about the software system, or (iii) accessing information about resources in the multi-tenant provider network which are used to run the software system; prompt, by the system design agent of the executing SDS code, the LLM with a second prompt to answer individual questions in the set of questions using the obtained data; receive answers to some questions in the set of questions from the LLM; determine that a particular question in the set of questions remains unanswered; request a response from the user to the particular question; receive an answer to the particular question from the user; prompt, by the system design agent of the executing SDS code, the LLM with a third prompt to respond with a proposed solution to the software development task, wherein the third prompt includes the answers received from the LLM and the answer to the particular question received from the user as well as at least some of the obtained data; receive a response to the software development task from the LLM; and send the response to the software development task to the user.
  2. 2 . The system of claim 1 , further comprising a chat-based interface configured to receive the description of the software development task from the user and output the response to the software development task to the user.
  3. 3 . The system of claim 1 , wherein the first prompt includes a limit on a maximum number of questions that can be included in the set of questions.
  4. 4 . A system comprising: a large language model (LLM) in a multi-tenant provider network; and a software development service (SDS) in the multi-tenant provider network, the SDS executing SDS code using one or more processors to cause the SDS to: receive, by a system design agent of the executing SDS code from a user, a description of a software development task to make a change to a software system; obtain, by one or more data-gathering context aggregators of the executing SDS code in response to receiving the description of the software development task to make the change to the software system, data associated with at least the software system from a data source by one or more of (i) accessing source code of the software system, (ii) accessing documentation about the software system, or (iii) accessing information about resources in the multi-tenant provider network which are used to run the software system; prompt, by the system design agent of the executing SDS code, the LLM to identify at least one aspect of implementing the change to the software system which requires clarification from the user, at least partly by providing the obtained data to the LLM and asking the LLM to identify a question for the user which remains unanswered by the obtained data; present the question to the user; receive an answer to the question from the user; prompt, by the system design agent of the executing SDS code, the LLM to respond to propose an implementation of the change to the software system at least partly based on the data associated with the software system and the answer received from the user; receive the proposed implementation of the change to the software system from the LLM; and cause display of the proposed implementation of the change to the software system to the user.
  5. 5 . The system of claim 4 , wherein to prompt the LLM to identify at least one aspect of implementing the change which requires clarification from the user, the SDS is to: prompt the LLM with a first prompt to generate a set of questions, wherein the first prompt includes the description of the software development task; receive the set of questions from the LLM; and prompt the LLM with a second prompt to answer questions in the set of questions using the obtained data, wherein the question for the user which remains unanswered is a question in the set of questions.
  6. 6 . The system of claim 5 , wherein the obtained data is also associated with a plurality of other software systems accessible to the user, and wherein to ask the LLM to identify the question for the user which remains unanswered by the obtained data, the SDS is to: prompt the LLM to indicate whether there is a consensus across answers to the question based on each software system represented in the obtained data including the software system and the other software systems; and receive, from the LLM, an indication that there is not consensus in answers to the question.
  7. 7 . The system of claim 5 , wherein to ask the LLM to identify a question for the user which remains unanswered by the obtained data, the SDS is to: prompt the LLM to provide a confidence score associated with an LLM-provided answer to the question for the software system; receive the confidence score from the LLM; and determine that the confidence score does not satisfy a threshold.
  8. 8 . The system of claim 5 , wherein the first prompt includes a limit on a maximum number of questions that can be included in the set of questions.
  9. 9 . The system of claim 8 , wherein the SDS is further configured to: prompt the LLM with a third prompt to identify an estimated number of questions related to design considerations to respond to the software development task; and receive the estimated number of questions from the LLM, wherein the limit is the estimated number of questions.
  10. 10 . The system of claim 5 , wherein an application including a chat-based interface provides an interface for the user, the description of the software development task received from the application, and the proposed implementation of the change to the software system sent to the application.
  11. 11 . The system of claim 5 , wherein the SDS is further configured to: prompt the LLM with a validation prompt to check whether the set of questions conforms with a response definition, the validation prompt including the response definition and the set of questions; and receive, from the LLM, an indication that the set of questions conforms with the response definition.
  12. 12 . The system of claim 4 , wherein the SDS is further configured to: prompt the LLM with a sanitization prompt to check whether the answer from the user includes objectionable material; and receive, from the LLM, an indication that the answer does not include objectionable material.
  13. 13 . A computer-implemented method comprising: receiving, by a system design agent implemented by a software development service (SDS) executing SDS code using one or more processors, from a user, a description of a software development task to make a change to a software system; obtaining, by one or more data-gathering context aggregators of the executing SDS code in response to receiving the description of the software development task to make the change to the software system, data associated with at least the software system from a data source by one or more of (i) accessing source code of the software system, (ii) accessing documentation about the software system, or (iii) accessing information about resources in a multi-tenant provider network which are used to run the software system; prompting, by the system design agent of the executing SDS code, a large language model (LLM) to identify at least one aspect of implementing the change to the software system which requires clarification from the user, at least partly by providing the obtained data to the LLM and asking the LLM to identify a question for the user which remains unanswered by the obtained data; presenting the question to the user; receiving an answer to the question from the user; prompting, by the system design agent of the executing SDS code, the LLM to respond to propose an implementation of the change to the software system at least partly based on the data associated with the software system and the answer received from the user; receiving the proposed implementation of the change to the software system from the LLM; and causing display of the proposed implementation of the change to the software system to the user.
  14. 14 . The computer-implemented method of claim 13 , wherein to prompt the LLM to identify at least one aspect of implementing the change which requires clarification from the user includes: prompting the LLM with a first prompt to generate a set of questions, wherein the first prompt includes the description of the software development task; receiving the set of questions from the LLM; and prompting the LLM with a second prompt to answer questions in the set of questions using the obtained data, wherein the question for the user which remains unanswered is a question in the set of questions.
  15. 15 . The computer-implemented method of claim 14 , wherein the obtained data is also associated with a plurality of other software systems accessible to the user, and wherein to ask the LLM to identify the question for the user which remains unanswered by the obtained data includes: prompting the LLM to indicate whether there is a consensus across answers to the question based on each software system represented in the obtained data including the software system and the other software systems; and receiving, from the LLM, an indication that there is not consensus in answers to the question.
  16. 16 . The computer-implemented method of claim 14 , wherein to ask the LLM to identify the question for the user which remains unanswered by the obtained data includes: prompting the LLM to provide a confidence score associated with an LLM-provided answer to the question for the software system; receiving the confidence score from the LLM; and determining that the confidence score does not satisfy a threshold.
  17. 17 . The computer-implemented method of claim 14 , wherein the first prompt includes a limit on a maximum number of questions that can be included in the set of questions.
  18. 18 . The computer-implemented method of claim 17 , further comprising: prompting the LLM with a third prompt to identify an estimated number of questions related to design considerations to respond to the software development task; and receiving the estimated number of questions from the LLM, wherein the limit is the estimated number of questions.
  19. 19 . The computer-implemented method of claim 14 , wherein an application including a chat-based interface provides an interface for the user, the description of the software development task received from the application, and the proposed implementation of the change to the software system sent to the application.
  20. 20 . The computer-implemented method of claim 13 , further comprising: prompting the LLM with a sanitization prompt to check whether the answer from the user includes objectionable material; and receiving, from the LLM, an indication that the answer does not include objectionable material.

Description

BACKGROUND Machine learning (ML) refers to a discipline by which computer systems can be trained to recognize patterns through repeated exposure to training data. The use of a trained model in production is often referred to as “inference,” during which the model receives new data that was not in its training data set and provides an output based on its learned parameters. In contrast to machine learning (ML), artificial intelligence (AI) refers to a human perception of a computer system as possessing a capability typically considered to require intelligence. Large language models (LLMs) are ML/AI systems designed to understand and generate human-like text. These models are trained using machine learning techniques, typically on vast amounts of text data from the internet, books, articles, and other sources. Often, LLMs use a type of neural network called a transformer to process and understand the patterns and structures of language. The underlying model of an LLM often consists of millions or even billions of model parameters, which are adjustable values that determine how the model behaves. The models are typically trained using a process called unsupervised learning, where they learn to predict the next word or sequence of words in a sentence based on the context provided. By doing so, LLMs develop an understanding of grammar, syntax, and semantics. As a result, LLMs are being increasingly deployed to aid in a variety of fields, including customer support, healthcare, language translation, education, software development, finance, and more. BRIEF DESCRIPTION OF DRAWINGS Various examples in accordance with the present disclosure will be described with reference to the drawings, in which: FIG. 1 depicts a high level overview of software development service and environment according to some examples. FIG. 2 depicts additional details of a software development service according to some examples. FIG. 3 depicts details of a system map context aggregator according to some examples. FIG. 4 depicts example templates and response definitions according to some examples. FIGS. 5 and 6 depict example operations of a validation agent according to some examples. FIGS. 7 and 8 depict example operations of an orchestrator agent according to some examples. FIG. 9 depicts example operation of a sanitization agent according to some examples. FIGS. 10 and 11 depict example operations of a system design agent according to some examples. FIG. 12 is a flow diagram illustrating operations of a method for generating solutions to a software development task according to some examples. FIGS. 13 through 15 depict example operations of a development task agent according to some examples. FIG. 16 is a flow diagram illustrating operations of a method for dividing a software development task into an action set according to some examples. FIGS. 17 and 18 depict example operations of an error resolution agent according to some examples. FIG. 19 is a flow diagram illustrating operations of a method for generating suggested resolutions to an error according to some examples. FIG. 20 depicts an interface controller of the software development service according to some examples. FIG. 21 illustrates a graphical user interface of the software development service according to some examples. FIGS. 22 through 26 illustrate a progression of changes to a software system with the software development service according to some examples. FIG. 27 is a flow diagram illustrating operations of a method for displaying a graphical user interface of a software development system according to some examples. FIG. 28 illustrates an example provider network environment according to some examples. FIG. 29 is a block diagram of an example provider network that provides a storage service and a hardware virtualization service to customers according to some examples. FIG. 30 is a block diagram illustrating an example computer system that can be used in some examples. DETAILED DESCRIPTION The present disclosure relates to an AI-powered developer productivity assistant that works alongside software developers to help them with different tasks such as designing, coding, and testing changes to software applications, and fixing errors or bugs in software applications. This productivity assistant can be provided as a software development service (SDS) in a cloud provider network and/or can be integrated into a development environment, in various implementations. The SDS supports a broad array of tasks, from small bug fixes to sweeping refactoring changes that impact multiple inter-dependent applications. Because it understands developers' intent, the SDS can also generate tests that validate the application's behavior in various failure modes, making it easier to deliver highly-available software to production quickly and reliably. As described above, the SDS can help developers fix errors, bugs, and vulnerabilities in a software program. Specifically, the SDS can identif