EP-4738101-A1 - GENERATIVE AI PROMPT CYCLE AND REFINEMENT FOR INDUSTRIAL AUTOMATION VISUALIZATION

EP4738101A1EP 4738101 A1EP4738101 A1EP 4738101A1EP-4738101-A1

Abstract

A human-machine interface (HMI) development system leverages a generative Al model to assist in development of HMI projects in accordance with specified functional requirements, which can be provided to the development system as intuitive natural language spoken or written text. The system can formulate and implement HMI project edits during design time based on analysis of this natural language design input. After the HMI project is deployed as a runtime application, the system can also receive and process natural language requests to modify the runtime HMI in accordance with described modification criteria.

Inventors

BRIANT, STEPHEN C.
BILLI-DURAN, Sharon
MILLER, MICHAEL B.
FUMAGALLI, Maurizio
KOWAL, STEVEN J.

Assignees

Rockwell Automation Technologies, Inc.

Dates

Publication Date: 20260506
Application Date: 20251023

Claims (15)

A system, comprising: a memory that stores executable components; and a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising: a user interface component configured to render a human-machine interface (HMI) development interface and to receive, via interaction with the HMI development interface, natural language design input that describes a functional requirement of an HMI project being developed using the system, wherein the HMI project defines display screens, layouts of graphical objects on the display screens, and communication links between the graphical objects and corresponding sources of data generated by an industrial automation system; a generative artificial intelligence (AI) component configured to, in response to determining, based on analysis of the natural language design input, that additional information will allow the generative AI component to determine an edit to the HMI project having a probability of satisfying the functional requirement that exceeds a threshold, generate a natural language response that prompts for the additional information, render the natural language response via the user interface component, and determine the edit to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information; and an HMI generation component configured to apply the edit to the HMI project.
The system of claim 1, wherein the HMI project is configured to, in response to execution on an HMI terminal device, read data from the sources of data and visualize the data via the graphical objects defined by the HMI project.
The system of claim 1 or 2, wherein the edit comprises at least one of creating the HMI project, creating a display screen of the display screens, setting a color scheme of the display screen, adding a graphical object to the display screen, setting a location or an orientation of the graphical object on the display screen, defining a communication link between an animation property of the graphical object and a data tag of an industrial controller, resizing the graphical object, defining an alarm condition that is to trigger rendering of an alarm, defining alarm text for the alarm, adding a graphical control to the display screen, defining a navigation scheme for navigating between the display screens, adding a definition of an industrial controller with which the HMI project will exchange data values, or generating a script to be executed in response to a condition specified by the natural language design input.
The system of one of claims 1 to 3, at least one of: wherein the training data comprises at least one of information defining industrial standards, technical specifics for respective types of industrial control applications, knowledge of different industrial verticals, information describing industrial best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for respective types of control applications, or customer-specific training data describing in-house HMI design preferences; wherein the natural language design input describes at least one of a visualization layout to be implemented by the display screens, a screen navigation requirement, industrial assets included in the industrial automation system, a type of manufacturing application performed by the industrial automation system, an alarm requirement, or a source of data that is to control an animation property of one of the graphical objects; and wherein the generative AI component is further configured to, in response to receipt of the natural language design input, formulate a prompt, directed to a generative AI model, designed to obtain a response from the generative AI model comprising information used by the generative AI component to determine the edit to be applied to the HMI project, wherein the prompt is generated based on analysis of the natural language design input and the content of one or more custom models.
The system of one of claims 1 to 4, wherein the user interface is configured to render, via the HMI development interface, selectable pre-composed natural language prompts, and in response to receiving, via interaction with HMI development interface, selection of a pre-composed prompt of the selectable pre-composed natural language prompts, submit the pre-composed prompt to the generative AI component as the natural language design input.
The system of claim 5, wherein the user interface component is further configured to receive, via interaction with the HMI development interface, a customization to the pre-composed prompt prior to submission of the pre-composed prompt to the generative AI component, and the customization customizes one or more parameters of the pre-composed prompt.
The system of one of claims 1 to 6, wherein the generative AI component is further configured to retrain the one or more custom models based on the edit, the natural language design input, and the additional information.
A method, comprising: rendering, by a system comprising a processor, a human-machine interface (HMI) development interface on a client device; receiving, by the system via interaction with the HMI development interface, natural language design input that describes a functional requirement of an HMI project being developed using the system, wherein the HMI project defines display screens, layouts of graphical objects on the display screens, and communication links between the graphical objects and corresponding sources of data generated by an industrial automation system; in response to determining, based on analysis of the natural language design input, that additional information will allow the system to determine an edit to the HMI project having a probability of satisfying the functional requirement that exceeds a threshold: generating, by the system, a natural language response that prompts for the additional information; rendering, by the system, the natural language response via the user interface component; and determining, by the system, the edit to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information; and applying, by the system, the edit to the HMI project.
The method of claim 8, wherein the HMI project is configured to, in response to execution on an HMI terminal device, read data from the sources of data and visualize the data via the graphical objects defined by the HMI project.
The method of claim 8 or 9, wherein the applying of the edit comprises at least one of creating the HMI project, creating a display screen of the display screens, setting a color scheme of the display screen, adding a graphical object to the display screen, setting a location or an orientation of the graphical object on the display screen, defining a communication link between an animation property of the graphical object and a data tag of an industrial controller, resizing the graphical object, defining an alarm condition that is to trigger rendering of an alarm, defining alarm text for the alarm, adding a graphical control to the display screen, defining a navigation scheme for navigating between the display screens, adding a definition of an industrial controller with which the HMI project will exchange data values, or generating a script to be executed in response to a condition specified by the natural language design input.
The method of one of claims 8 to 10, at least one of: wherein the training data comprises at least one of information defining industrial standards, technical specifics for respective types of industrial control applications, knowledge of different industrial verticals, information describing industrial best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for respective types of control applications, or customer-specific training data describing in-house HMI design preferences; wherein the natural language design input describes at least one of a visualization layout to be implemented by the display screens, a screen navigation requirement, industrial assets included in the industrial automation system, a type of manufacturing application performed by the industrial automation system, an alarm requirement, or a source of data that is to control an animation property of one of the graphical objects; and further comprising, in response to the receiving of the natural language design input, formulating, by the system, a prompt, directed to a generative AI model, designed to obtain a response from the generative AI model comprising information used by the system to determine the edit to be applied to the HMI project, wherein the prompt is generated based on analysis of the natural language design input and the content of one or more custom models.
The method of one of claims 8 to 11, further comprising: rendering, by the system, via the HMI development interface, selectable pre-composed natural language prompts, and in response to receiving, via interaction with HMI development interface, selection of a pre-composed prompt of the selectable pre-composed natural language prompts, submitting, by the system, the pre-composed prompt to the generative AI component as the natural language design input.
The method of claim 12, further comprising receiving, by the system via interaction with the HMI development interface, a customization to the pre-composed prompt prior to submission of the pre-composed prompt to the generative AI component, wherein the customization customizes one or more parameters of the pre-composed prompt.
A non-transitory computer-readable medium having stored thereon instructions that, in response to execution, cause a human-machine interface (HMI) development system comprising a processor to perform operations, the operations comprising: rendering a human-machine interface (HMI) development interface on a client device; receiving, via interaction with the HMI development interface, natural language design input that describes a functional requirement of an HMI project being developed using the system, wherein the HMI project defines display screens, layouts of graphical objects on the display screens, and communication links between the graphical objects and corresponding sources of data generated by an industrial automation system; in response to determining, based on analysis of the natural language design input, that additional information will allow the HMI development system to determine an edit to the HMI project having a probability of satisfying the functional requirement that exceeds a threshold: generating a natural language response that prompts for the additional information; rendering the natural language response via the user interface component; and determining the edit to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information; and applying the edit to the HMI project.
The non-transitory computer-readable medium of claim 14, at least one of: wherein the applying of the edit comprises at least one of creating the HMI project, creating a display screen of the display screens, setting a color scheme of the display screen, adding a graphical object to the display screen, setting a location or an orientation of the graphical object on the display screen, defining a communication link between an animation property of the graphical object and a data tag of an industrial controller, resizing the graphical object, defining an alarm condition that is to trigger rendering of an alarm, defining alarm text for the alarm, adding a graphical control to the display screen, defining a navigation scheme for navigating between the display screens, adding a definition of an industrial controller with which the HMI project will exchange data values, or generating a script to be executed in response to a condition specified by the natural language design input; and wherein the natural language design input describes at least one of a visualization layout to be implemented by the display screens, a screen navigation requirement, industrial assets included in the industrial automation system, a type of manufacturing application performed by the industrial automation system, an alarm requirement, or a source of data that is to control an animation property of one of the graphical objects.

Description

TECHNICAL FIELD The subject matter disclosed herein relates generally to industrial automation systems, and, for example, to development and deployment of industrial human-machine interfaces (HMIs). BACKGROUND ART Industrial human-machine interfaces, or HMIs, comprise a computer terminal with display capabilities that executes an HMI runtime application, which defines the display screens that are presented to the operator of an industrial automation system, the navigation structure for navigating between the display screens, and the data links or bindings between the graphical elements and corresponding data tags in the controller's data table. HMI developers typically design these aspects of an HMI using an HMI development platform. These HMI development platforms support a graphical and menu-driven development workflow in which the developer selects graphical display and control elements from a library of elements for inclusion on each display interface, and manipulates these selected elements - e.g., via drag-and-drop interactions - on a mock-up of the display interface to yield a desired layout. BRIEF DESCRIPTION The following presents a simplified summary in order to provide a basic understanding of some aspects described herein. This summary is not an extensive overview nor is intended to identify key/critical elements or to delineate the scope of the various aspects described herein. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. In one or more embodiments, a system is provided, comprising a user interface component configured to render a human-machine interface (HMI) development interface and to receive, via interaction with the HMI development interface, natural language design input that describes a functional requirement of an HMI project being developed using the system, wherein the HMI project defines display screens, layouts of graphical objects on the display screens, and communication links between the graphical objects and corresponding sources of data generated by an industrial automation system; a generative artificial intelligence (AI) component configured to, in response to determining, based on analysis of the natural language design input, that additional information will allow the generative AI component to determine an edit to the HMI project having a probability of satisfying the functional requirement that exceeds a threshold, generate a natural language response that prompts for the additional information, render the natural language response via the user interface component, and determine the edit to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information; and an HMI generation component configured to apply the edit to the HMI project. Also, one or more embodiments provide a method, comprising rendering, by a system comprising a processor, a human-machine interface (HMI) development interface on a client device; receiving, by the system via interaction with the HMI development interface, natural language design input that describes a functional requirement of an HMI project being developed using the system, wherein the HMI project defines display screens, layouts of graphical objects on the display screens, and communication links between the graphical objects and corresponding sources of data generated by an industrial automation system; in response to determining, based on analysis of the natural language design input, that additional information will allow the system to determine an edit to the HMI project having a probability of satisfying the functional requirement that exceeds a threshold: generating, by the system, a natural language response that prompts for the additional information; rendering, by the system, the natural language response via the user interface component; and determining, by the system, the edit to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information; and applying, by the system, the edit to the HMI project. Also, according to one or more embodiments, a non-transitory computer-readable medium is provided having stored thereon instructions that, in response to execution, cause a human-machine interface (HMI) development system to perform operations, the operations comprising rendering a human-machine interface (HMI) development interface on a client device; receiving, via interaction with the HMI development interface, natural language design input that describes a functional requirement of an HMI project being developed using the system, wherein the HMI project defines display screens, layouts of graphical objects on the display screens, and communication links between the graphical objects and corresponding sourc