US-12619424-B2 - Robotic script generation based on process variation detection

Abstract

Techniques for generating Robotic Scripts via Process Variation Detection are described. In one example, captured process steps related to an activity performed in an application may be received. Variations of the process steps are then determined by training a first Artificial Neural Network (ANN) with the captured process steps. A set of the process steps for performing the activity, may then be determined based on the determined variations of the process steps. Robotic scripts may be generated using the determined set of process steps to perform the activity.

Inventors

• Ravi Ramamurthy
• RAMGOPAL M. CILLANKI

Assignees

• EPIANCE SOFTWARE PVT. LTD.

Dates

Publication Date

20260505

Application Date

20180612

Priority Date

20180424

Claims (20)

1. 1 . A system comprising: a processor; and a memory communicatively coupled to the processor, wherein the memory is capable of executing a plurality of modules stored in the memory, and wherein the plurality of modules comprises: a receiving module to receive captured process steps related to an activity performed in an enterprise resource planning (ERP) application, wherein the captured process steps comprise multiple sequences of graphical user interface (GUI) interactions carried out by users for performing the activity in the ERP application, and wherein the GUI interactions refer to human interface device actions performed on a GUI to execute the activity; a processing module to determine multiple variations of process steps for performing the activity by applying a first Artificial Neural Network (ANN) to the captured process steps, wherein each variation includes a combination of the process steps such that the activity can be performed by executing any one of the multiple variations, and wherein the first ANN determines a category to which the process steps belong to thereby connecting the process steps to the activity; an optimization module to determine an optimized set of the process steps for performing the activity based on the determined variations of the process steps, wherein the optimized set of process steps includes process steps selected from at least one of the multiple variations such that the activity is performed in an manner with a reduced amount of processing resources; and a script generation module to: compare screenshots of the ERP application taken at different time intervals to identify a control field location, control field type, and control field name at which a next user action occurs; and upon identifying the control field location, control field type, and control field name at which the next user action occurs, generate a software robot to automate a process of performing the activity associated with the next user action using the determined optimized set of process steps, wherein the generated software robot, upon execution, mimics the GUI interactions by executing the optimized set of the process steps for performing the activity.
2. 2 . The system of claim 1 , wherein the script generation module generates the software robot automatically.
3. 3 . The system of claim 1 , wherein the script generation module generates the software robot manually in a simulated environment.
4. 4 . The system of claim 1 , wherein the captured process steps are fed to the first ANN in the form of Extensible Markup Language (XML) files and/or hash codes.
5. 5 . The system of claim 1 , wherein the first ANN is a Recurrent Neural Network (RNN), and wherein the RNN is a Long Short-Term Memory (LSTM) network.
6. 6 . The system of claim 1 , comprising a classification module, wherein the receiving module is to receive an input document; the classification module is to classify the input document using a second ANN; the processing module is to determine the input document that corresponds to the activity based on the classification; and the robotic script generation module is to automatically fill a form related to the input document by executing the set of process steps upon determining that the input document corresponds to the activity.
7. 7 . The system of claim 6 , wherein the input document is a text document and/or an image document.
8. 8 . The system of claim 7 , wherein the second ANN is a Feed Forward Neural Network, and wherein the Feed Forward Neural Network is a Convolutional Neural Network (CNN) or a Deep Auto Encoder.
9. 9 . The system of claim 8 , wherein the classification module is configured to: classify the text document using the Deep Auto Encoder; and classify the image document using the CNN.
10. 10 . A computer-implemented method comprising: receiving captured process steps related to an activity performed in an enterprise resource planning (ERP) application, wherein the captured process steps comprise multiple sequences of graphical user interface (GUI) interactions carried out by users for performing the activity in the ERP application, and wherein the GUI interactions refer to human interface device actions performed on a GUI to execute the activity; determining multiple variations of process steps for performing the activity by applying a first Artificial Neural Network (ANN) to the captured process steps, wherein each variation includes a combination of the process steps such that the activity can be performed by executing any one of the multiple variations, and wherein the first ANN determines a category to which the process steps belong to thereby connecting the process steps to the activity; determining an optimized set of the process steps for performing the activity based on the determined variations of the process steps, wherein the optimized set of process steps includes process steps selected from at least one of the multiple variations such that the activity is performed in an manner with a reduced amount of processing resources; and comparing screenshots of the ERP application taken at different time intervals to identify a control field location, control field type, and control field name at which a next user action occurs; and upon identifying the control field location, control field type, and control field name at which the next user action occurs, generating a software robot to automate a process of performing the activity associated with the next user action using the determined optimized set of process steps, wherein the generated software robot, upon execution, mimics the GUI interactions by executing the optimized set of the process steps for performing the activity.
11. 11 . The computer-implemented method of claim 10 , wherein the software robot is generated automatically, or manually in a simulated environment.
12. 12 . The computer-implemented method of claim 10 , wherein the captured process steps are fed to the first ANN in the form of Extensible Markup Language (XML) files and/or hash codes.
13. 13 . The computer-implemented method of claim 10 , wherein the first ANN is a Recurrent Neural Network (RNN), and wherein the RNN is a Long Short-Term Memory (LSTM) network.
14. 14 . The computer-implemented method of claim 10 , wherein generating the software robot to perform the activity comprises: receiving an input document; classifying the input document using a second ANN; determining the input document corresponds to the activity based on the classification; and automatically filling a form related to the input document by executing the set of process steps upon determining that the input document corresponds to the activity.
15. 15 . The computer-implemented method of claim 14 , wherein the second ANN is a Feed Forward Neural Network, and wherein the Feed Forward Neural Network is a Convolutional Neural Network (CNN) or a Deep Auto Encoder.
16. 16 . The computer-implemented method of claim 15 , wherein classifying the input document comprises: classifying the input document using the Deep Auto Encoder upon determining that the input document is a text document; and classifying the input document using the CNN upon determining that the input document is an image document.
17. 17 . A non-transitory machine-learning medium storing instructions executable by a processing resource to: receive captured process steps related to an activity performed in an enterprise resource planning (ERP) application, wherein the captured process steps comprise multiple sequences of graphical user interface (GUI) interactions carried out by users for performing the activity in the ERP application, and wherein the GUI interactions refer to human interface device actions performed on a GUI to execute the activity; determine multiple variations of process steps for performing the activity by applying a first Artificial Neural Network (ANN) to the captured process steps, wherein each variation includes a combination of the process steps such that the activity can be performed by executing any one of the multiple variations, and wherein the first ANN determines a category to which the process steps belong to thereby connecting the process steps to the activity; determine an optimized set of the process steps for performing the activity based on the determined variations of the process steps, wherein the optimized set of process steps includes process steps selected from at least one of the multiple variations such that the activity is performed in an manner with a reduced amount of processing resources; and compare screenshots of the ERP application taken at different time intervals to identify a control field location, control field type, and control field name at which a next user action occurs; and upon identifying the control field location, control field type, and control field name at which the next user action occurs, generate a software robot to automate a process of performing the activity associated with the next user action using the determined optimized set of process steps, wherein the generated software robot, upon execution, mimics the GUI interactions by executing the optimized set of the process steps for performing the activity.
18. 18 . The non-transitory machine-readable medium of claim 17 , wherein the software robot is generated automatically, or manually in a simulated environment.
19. 19 . The non-transitory machine-readable medium of claim 17 , wherein the first ANN is a Long Short-Term Memory (LSTM) Network.
20. 20 . The non-transitory machine-readable medium of claim 17 , wherein generating the software robot to perform the activity comprises instructions to; receive an input document; classify the input document using a second ANN; determining the input document corresponds to the activity based on the classification; and automatically fill a form related to the input document by executing the set of process steps upon determining that the input document corresponds to the activity.

Description

RELATED APPLICATIONS Benefit is claimed under 35 U.S.C. 119(a)-(d) to Foreign Application Serial No. 201841015557 filed in India entitled “ROBOTIC SCRIPT GENERATION BASED ON PROCESS VARIATION DETECTION”, on Apr. 24, 2018, by Epiance Software Pvt. Ltd., which is herein incorporated in its entirety by reference for all purposes. BACKGROUND Currently, various software applications have interfaces designed to accomplish certain activities by following a sequence of Graphical User Interface (GUI) interactions. Activities may include business process activities like collecting and organizing data from databases, filling application forms, digitalizing offline forms by manual data entry, and the like. Such activities may be carried out manually by a user of the software application by following a sequence of GUI interactions. Some of these activities may be repetitive in nature but may be unavoidable. In such a scenario, using human users to manually perform these interactions may be time consuming and error prone. However, using human users may be unavoidable as most of these process steps are heavily reliant on the decision-making abilities of the user. BRIEF DESCRIPTION OF THE DRAWINGS Examples are described in the following detailed description and in reference to the drawings, in which: FIG. 1 is a block diagram illustrating an example robotic script generation system connected to a plurality of devices via a network; FIG. 2A is a block diagram of an example Artificial Neural Network (ANN) trained for determining process variations; FIG. 2B is an example illustration of a Recurrent Neural Network trained for determining process variations; FIG. 2C is an example illustration of a Feed Forward Neural Network trained for performing process discovery; FIG. 3A depicts an example schematic diagram illustrating an activity performed in target business application via a robotic process using robotic script generation system; FIG. 3B depicts another example schematic diagram illustrating an activity performed in target business application via a robotic process using a robotic script generation system; FIG. 4 is a flow diagram for generating robotic scripts to automatically fill input documents; FIG. 5 is a block diagram illustrating an example robotic script generation platform including the robotic script generation system of FIG. 1; FIG. 6 is an example flow diagram of a method of generating a robotic script by process variation detection using ANNs; FIG. 7 is an example flow diagram of the method of generating a robotic script for automatically filling input documents; FIG. 8 is an example flow diagram of a process for predicting a next step to be performed in an activity; FIG. 9A depicts example screenshots of a GUI to determine user interactions; FIG. 9B is an example portion of the GUI of FIG. 9A depicting focus areas to determine the user interactions; FIG. 10 is an example flow diagram for visual user interface identification using Artificial Neural Networks; FIG. 11 is an example flow diagram for visual user interface identification using pattern recognition techniques; FIG. 12 is an example flow diagram of a change management process; FIG. 13 is an example high-level architecture of the robotic script generation system using ANNs capable of performing various functions; and FIG. 14 is a block diagram of an example system instructions in a machine-readable storage medium for robotic script generation. DETAILED DESCRIPTION The Robotic Script Generation System may capture user interactions (e.g., process steps) in a Graphical User Interface (GUI) based application. The captured process steps may relate to user actions performed in the GUI to execute an activity in an application. The captured process steps may then be used in training a first Artificial Neural Network (ANN) to determine variations of process steps for performing the activity. Based on the determined variations, a set of process steps may then be determined. Further, based on the determined set of process steps, robotic scripts may be generated for performing the activity. Furthermore, the robotic scripts, upon execution, may automatically execute the set of process steps to perform the activity in the software application. Examples described herein may include various sub-systems that may or may not be necessary and may be assembled as needed for any given requirement. Further, the term “user interactions” refers to human interface device (e.g., mouse, keyboard, trackpad and so on) actions performed by the user or a robot. The actions may be performed on displayed “textual and visual information”, which may be graphical user interface (GUI) elements, character-based interfaces and so on. The terms “user interactions”, “user actions”, and “process steps” may be used interchangeably throughout this document. Furthermore, example target business process software applications may include SAP® applications, PeopleSoft® applications, SalesForce® app