CN-121996225-A - Generated AI hint loops and refinements for industrial automation visualization

CN121996225ACN 121996225 ACN121996225 ACN 121996225ACN-121996225-A

Abstract

Generated AI hint loops and refinements for industrial automation visualization are provided. The human-machine interface HMI development system utilizes a generative AI model to facilitate development of HMI projects based on specified functional requirements that can be provided to the development system as intuitive natural language spoken or written text. The system can formulate and implement HMI project edits during design time based on analysis of the natural language design input. After the HMI project is deployed as a runtime application, the system can also receive and process natural language requests to modify the runtime HMI according to the described modification criteria.

Inventors

Stephen c bryant
Sha Long.bili-dulan
MILLER MICHAEL B
Maurizio Fumagali
KOWAL STEVEN J

Assignees

罗克韦尔自动化技术公司

Dates

Publication Date: 20260508
Application Date: 20251030
Priority Date: 20241101

Claims (20)

1. A system, comprising: A memory storing executable components, and A processor operatively coupled to the memory, the processor executing the executable component, the executable component comprising: A user interface component configured to present a human machine interface, HMI, development interface and receive, via interaction with the HMI development interface, natural language design input describing functional requirements for developing an HMI project using the system, wherein the HMI project defines a display screen, a layout of graphical objects on the display screen, and a communication link between the graphical objects and corresponding data sources generated by an industrial automation system; a generated artificial intelligence AI component configured to, in response to determining based on analysis of the natural language design input that additional information will enable the generated AI component to determine edits to the HMI item having a likelihood of meeting the functional requirement exceeding a threshold, Generating a natural language response prompting the additional information, Presenting the natural language response via the user interface component, and Determining edits to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information, and An HMI generation component configured to apply the edits to the HMI project.
2. The system of claim 1, wherein the HMI item is configured to read data from the data source and visualize the data via the graphical object defined by the HMI item in response to execution on an HMI terminal device.
3. The system of claim 1, wherein the editing comprises at least one of creating the HMI item, creating a display screen in the display screen, setting a color scheme of the display screen, adding a graphical object to the display screen, setting a position or orientation of the graphical object on the display screen, defining a communication link between an animation attribute of the graphical object and a data tag of an industrial controller, resizing the graphical object, defining an alert condition for a presentation of an alert to be triggered, defining alert text for the alert, adding a graphical control to the display screen, defining a navigation scheme for navigating between the display screens, adding a definition of an industrial controller with which the HMI item is to exchange data values, or generating a script to be executed in response to a condition specified by the natural language design input.
4. The system of claim 1, wherein the training data includes at least one of information defining industry standards, technical details for respective types of industrial control applications, knowledge of different industry vertical fields, information describing industry best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for respective types of control applications, or customer-specific training data describing internal HMI design preferences.
5. The system of claim 1, wherein the natural language design input describes at least one of a visual layout to be implemented by the display screen, a screen navigation requirement, an industrial asset included in the industrial automation system, a type of manufacturing application executed by the industrial automation system, an alarm requirement, or a data source to control an animation property of one of the graphical objects.
6. The system of claim 1, wherein the generative AI component is further configured to formulate a hint for a generative AI model in response to receiving the natural language design input, the hint designed to obtain a response from the generative AI model, the response including information used by the generative AI component to determine the edits to be applied to the HMI project, wherein the hint is generated based on analysis of the natural language design input and content of one or more custom models.
7. The system of claim 1, wherein the user interface component is configured to: Presenting selectable pre-written natural language cues via the HMI development interface, and In response to receiving a selection of a pre-written prompt of the selectable pre-written natural language prompts via interaction with the HMI development interface, submitting the pre-written prompt to the generative AI component as the natural language design input.
8. The system of claim 7, wherein, The user interface component is further configured to receive customization of the pre-written prompts via interaction with the HMI development interface prior to submitting the pre-written prompts to the generated AI component, and The customization is performed on one or more parameters of the pre-written prompts.
9. The system of claim 1, wherein the generative AI component is further configured to retrain the one or more custom models based on the editing, the natural language design input, and the additional information.
10. A method, comprising: presenting, by a system including a processor, a human-machine interface (HMI) development interface on a client device; Receiving, by the system via interaction with the HMI development interface, a natural language design input describing functional requirements for developing an HMI project using the system, wherein the HMI project defines a display screen, a layout of graphical objects on the display screen, and a communication link between the graphical objects and corresponding data sources generated by an industrial automation system; Determining additional information in response to analysis based on the natural language design input will enable the system to determine edits of the HMI item having a likelihood of meeting the functional requirement exceeding a threshold: generating, by the system, a natural language response prompting the additional information; Presenting, by the system, a natural language response via a user interface component, and Determining, by the system, edits to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information, and The editing is applied to the HMI project by the system.
11. The method of claim 10, wherein the HMI item is configured to read data from the data source and visualize the data via the graphical object defined by the HMI item in response to execution on an HMI terminal device.
12. The method of claim 10, wherein applying the edits includes at least one of creating the HMI item, creating a display screen in the display screen, setting a color scheme of the display screen, adding a graphical object to the display screen, setting a position or orientation of the graphical object on the display screen, defining a communication link between an animation property of the graphical object and a data tag of an industrial controller, resizing the graphical object, defining an alert condition to trigger presentation of an alert, defining alert text of the alert, adding a graphical control to the display screen, defining a navigation scheme for navigating between the display screens, adding a definition of an industrial controller with which the HMI item is to exchange data values, or generating a script to be executed in response to a condition specified by the natural language design input.
13. The method of claim 10, wherein the training data includes at least one of information defining industry standards, technical details for respective types of industrial control applications, knowledge of different industry vertical fields, information describing industry best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for respective types of control applications, or customer-specific training data describing internal HMI design preferences.
14. The method of claim 10, wherein the natural language design input describes at least one of a visual layout to be implemented by the display screen, a screen navigation requirement, an industrial asset included in the industrial automation system, a type of manufacturing application executed by the industrial automation system, an alarm requirement, or a data source to control an animation property of one of the graphical objects.
15. The method of claim 10, further comprising, in response to receiving the natural language design input, formulating, by the system, a prompt for a generative AI model, the prompt designed to obtain a response from the generative AI model, the response including information used by the system to determine the edits to be applied to the HMI project, wherein the prompt is generated based on an analysis of the natural language design input and content of one or more custom models.
16. The method of claim 10, further comprising: presenting, by the system via the HMI development interface, selectable pre-written natural language cues, and Responsive to receiving a selection of a pre-written prompt of the selectable pre-written natural language prompts via interaction with the HMI development interface, submitting, by the system, the pre-written prompt to the generated AI component as the natural language design input.
17. The method of claim 16, further comprising receiving, by the system, customization of the pre-written prompts via interaction with the HMI development interface prior to submitting the pre-written prompts to the generated AI component, wherein the customization customizes one or more parameters of the pre-written prompts.
18. A non-transitory computer-readable medium having instructions stored thereon that, in response to execution, cause a human-machine interface HMI development system comprising a processor to perform operations comprising: presenting a human machine interface HMI development interface on a client device; Receiving, via interaction with the HMI development interface, a natural language design input describing functional requirements for developing an HMI project using the system, wherein the HMI project defines a display screen, a layout of graphical objects on the display screen, and a communication link between the graphical objects and corresponding data sources generated by an industrial automation system; determining additional information in response to analysis based on the natural language design input will enable the HMI development system to determine edits to the HMI project having a likelihood of meeting the functional requirement exceeding a threshold: generating a natural language response prompting the additional information; Presenting the natural language response via a user interface component, and Determining edits to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information, and The editing is applied to the HMI project.
19. The non-transitory computer-readable medium of claim 18, wherein applying the editing comprises at least one of creating the HMI item, creating a display screen in the display screen, setting a color scheme for the display screen, adding a graphical object to the display screen, setting a position or orientation of the graphical object on the display screen, defining a communication link between an animation attribute of the graphical object and a data tag of an industrial controller, resizing the graphical object, defining an alert condition to trigger presentation of an alert, defining alert text for the alert, adding a graphical control to the display screen, defining a navigation scheme for navigating between the display screens, adding a definition of an industrial controller with which the HMI item is to exchange data values, or generating a script to be executed in response to a condition specified by the natural language design input.
20. The non-transitory computer readable medium of claim 18, wherein the natural language design input describes at least one of a visual layout to be implemented by the display screen, a screen navigation requirement, an industrial asset included in the industrial automation system, a type of manufacturing application executed by the industrial automation system, an alarm requirement, or a data source to control an animation property of one of the graphical objects.

Description

Generated AI hint loops and refinements for industrial automation visualization Technical Field The subject matter disclosed herein relates generally to industrial automation systems and, for example, to the development and deployment of industrial human-machine interfaces (HMI). Background An industrial human-machine interface or HMI includes a computer terminal with display capabilities that executes an HMI runtime application that defines display screens presented to operators of industrial automation systems, navigation structures for navigating between the display screens, and data links or bindings between graphic elements and corresponding data tags in a data table of a controller. HMI developers typically use an HMI development platform to design these aspects of an HMI. These HMI development platforms support graphic and menu driven development workflows in which a developer selects graphic display and control elements from a library of elements to include on each display interface and manipulates these selected elements, e.g., via drag-and-drop interactions, on a prototype graph of the display interface to produce a desired layout. Disclosure of Invention The following presents a simplified summary in order to provide a basic understanding of some aspects described herein. This summary is not an extensive overview nor is intended to identify key/critical elements or to delineate the scope of the various aspects described herein. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. In one or more embodiments, a system is provided that includes a user interface component configured to present a Human Machine Interface (HMI) development interface and receive natural language design input describing functional requirements for developing an HMI project using the system via interaction with the HMI development interface, wherein the HMI project defines a display screen, a layout of graphical objects on the display screen, and a communication link between the graphical objects and corresponding data sources generated by an industrial automation system, a generative Artificial Intelligence (AI) component configured to, in response to determining additional information based on analysis of the natural language design input, enable the generative AI component to determine edits of the HMI project that have a likelihood of meeting the functional requirements that exceed a threshold, generate a natural language response prompting the additional information, present the natural language response via the user interface component, and determine edits to be applied to the HMI project based on analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information, and an HMI generation component configured to apply the edits to the HMI project. Further, one or more embodiments provide a method comprising presenting, by a system comprising a processor, a Human Machine Interface (HMI) development interface on a client device, receiving, by the system, natural language design input describing functional requirements for developing an HMI project using the system via interaction with the HMI development interface, wherein the HMI project defines a display screen, a layout of graphical objects on the display screen, and a communication link between the graphical objects and corresponding data sources generated by an industrial automation system, determining, in response to analysis of the natural language design input, additional information would enable the system to determine edits to the HMI project that have a likelihood of meeting the functional requirements that exceeds a threshold, generating, by the system, a natural language response prompting the additional information, presenting, by the system, the natural language response via a user interface component, and determining, by the system, edits to be applied to the HMI project based on the analysis of the natural language design input, content of one or more custom models trained with training data, and the additional information, and applying, by the system, to the HMI project. Further, in accordance with one or more embodiments, a non-transitory computer-readable medium is provided having instructions stored thereon that, in response to execution, cause a Human Machine Interface (HMI) development system to perform operations including presenting a Human Machine Interface (HMI) development interface on a client device, receiving natural language design input describing functional requirements for developing an HMI project using the system via interaction with the HMI development interface, wherein the HMI project defines a display screen, a layout of graphical objects on the display screen, and a communication link between the graphical objects and corresponding data sources generated by an industrial au