US-12619458-B2 - Interruption detection during automated workflow

US12619458B2US 12619458 B2US12619458 B2US 12619458B2US-12619458-B2

Abstract

Systems and methods are provided for detecting an interruption during an automated workflow. An automated workflow may comprise a series of actions to be performed by or with the assistance of a computer. A workflow manager executes a workflow by progressing through a series of workflow states according to instructions associated with the workflow. When the workflow advances to a new state, an interruption detection engine determines whether the state contains an interruption by examining one or more attributes of the workflow state and/or the user interface associated therewith. An interruption detecting engine may examine a document object model and/or utilize computer vision to determine whether an interruption has occurred. When an interruption is detected, a workflow is paused until the interruption is resolved, such as by a user providing a required input. After an interruption has been resolved, the resumes and continues until completion of the workflow.

Inventors

Micheal Dunn
Nabeel SHAHZAD
Sri Raghu Malireddi
Jonathan W. LIN
Olutayo FALASE
Deyuan KE
Joann T. LEE
Shrey Shah

Assignees

MICROSOFT TECHNOLOGY LICENSING, LLC

Dates

Publication Date: 20260505
Application Date: 20220421

Claims (20)

1 . A system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations comprising: initiating an automated workflow, the automated workflow comprising a plurality of workflow states, the initiating comprising loading an indication of a workflow state into the memory to facilitate performing the operations to automate the workflow; detecting a change from a first workflow state to a second workflow state; extracting a document object model (DOM) of the second workflow state; analyzing the DOM to detect a first DOM element comprising markup content; programmatically detecting, based on the markup content and one or more attributes of the first DOM element, a workflow interruption requiring a user input; classifying, based on the one or more attributes of the first DOM element, a type of workflow interruption by comparing the one or more attributes to DOM attributes associated with known workflow interruptions; modifying execution of the automated workflow based on the classified type of workflow interruption, including pausing the automated workflow and storing an indication of the paused workflow state in the memory; causing presentation of a representation of the second workflow state to a user; receiving an indication that the user input has been received; and based on receiving the indication, accessing from the memory the indication of the paused workflow state and resuming the automated workflow.
2 . The system of claim 1 , wherein the automated workflow is associated with a set of workflow instructions, the workflow instructions comprising a series of actions for proceeding through the plurality of workflow states.
3 . The system of claim 1 , wherein the workflow interruption comprises a CAPTCHA request.
4 . The system of claim 3 , wherein the user input has been received when the CAPTCHA request has been completed.
5 . The system of claim 1 , wherein the first DOM element comprises an iframe, and the markup content includes a keyword.
6 . The system of claim 5 , wherein the keyword indicates a “CAPTCHA” interruption.
7 . The system of claim 1 , wherein the known workflow interruptions include a prior workflow interruption; wherein the first DOM element is associated with a visual feature of a user interface presented during the second workflow state; wherein classifying the type of workflow interruption further comprises: identifying the visual feature; and determining a degree of similarity between the visual feature and a visual feature associated with the prior workflow interruption; wherein the degree of similarity is represented by a visual feature score, and wherein the type of workflow interruption is determined based on the visual feature score exceeding a threshold.
8 . The system of claim 7 , wherein the set of operations further comprises: determining pixel intensities of the first DOM element or a second DOM element of the DOM of the second workflow state; and generating a pixel intensity score indicating a degree of similarity between the pixel intensities of the first or second DOM elements and pixel intensities associated with the prior workflow interruption; and classifying the type of workflow interruption based on a combination of the visual feature score and the pixel intensity score.
9 . The system of claim 8 , wherein the pixel intensity score further indicates a first workflow-interruption type to which the pixel intensities of the second workflow state are the most similar, and wherein the visual features score further indicates a second workflow-interruption type to which the visual feature of the second workflow state is the most similar.
10 . The system of claim 9 , wherein determining that the second workflow state includes a workflow interruption comprises: generating a computer-vision score based on the pixel-intensity score and the visual-feature score; if the first workflow-interruption type and the second workflow-interruption type are the same, comparing the computer-vision score to a first threshold and determining that the computer-vision score exceeds the first threshold; and if the first workflow-interruption type and the second workflow-interruption type are not the same, comparing the computer-vision score to a second threshold and determining that the computer-vision score exceeds the first threshold, wherein the second threshold is higher than the first threshold.
11 . The system of claim 1 , wherein classifying the type of workflow interruption comprises comparing the one or more attributes of the first DOM element to DOM attributes associated with known workflow interruptions stored in interruption detection data.
12 . A computer-implemented method for detecting a workflow interruption, the method comprising: initiating an automated workflow, the automated workflow comprising a plurality of workflow states; detecting a change from a first workflow state to a second workflow state; extracting a document object model (DOM) of the second workflow state; analyzing the DOM to detect a DOM element comprising markup content; programmatically detecting, based on the markup content and one or more attributes of the DOM element, a workflow interruption requiring a user input; classifying, based on the one or more attributes of the DOM element, a type of workflow interruption by comparing the one or more attributes to DOM attributes associated with known workflow interruptions; modifying execution of the automated workflow based on the classified type of workflow interruption, including pausing the automated workflow; causing presentation of the second workflow state to a user; receiving an indication that the user input has been received; and based on receiving the indication, resuming the automated workflow.
13 . The computer-implemented method of claim 12 , wherein the workflow interruption comprises a CAPTCHA request.
14 . The computer-implemented method of claim 13 , wherein examining the attribute of the second workflow state comprises: determining, based on the indication of the DOM stored in the database, that the second workflow state comprises an iframe; and identifying the keyword in a markup of the iframe, wherein the keyword is “CAPTCHA”.
15 . The computer-implemented method of claim 13 , wherein examining the attribute of the second workflow state comprises: identifying a visual feature of a user interface presented during the second workflow state; and determining a degree of similarity between the visual feature of the user interface associated with the second workflow state and a visual feature associated with a prior workflow interruption.
16 . The computer-implemented method of claim 13 , wherein examining the attribute of the second workflow state comprises: determining pixel intensities of the user interface associated with the second workflow state; and determining a degree of similarity between the pixel intensities of the user interface and pixel intensities of a prior workflow interruption.
17 . The computer-implemented method of claim 14 , wherein examining the attribute of the second workflow state comprises further comprises: identifying a visual feature of a user interface presented during the second workflow state; and determining a degree of similarity between the visual feature of the user interface associated with the second workflow state and a visual feature associated with a prior workflow interruption.
18 . The computer-implemented method of claim 17 , wherein determining the degree of similarity comprises generating a visual feature score.
19 . The computer-implemented method of claim 18 , wherein determining, based on the examining, that the second workflow state includes a workflow interruption comprises comparing the visual feature score to a visual feature threshold.
20 . A non-transitory computer-storage medium encoding computer-executable instructions that, when executed by at least one processor, perform a method comprising: initiating an automated workflow, the automated workflow comprising a plurality of workflow states; detecting a change from a first workflow state to a second workflow state; identifying a visual feature of a user interface presented during the second workflow state; storing an indication of the visual feature in a memory; executing a computer vision program to process the indication of the visual feature by: accessing from the memory an indication of and a prior visual feature associated with a prior workflow interruption; generating, based on the indication of the visual feature stored in the memory, a visual feature score indicating a degree of similarity between the visual feature of the user interface associated with the second workflow state and the prior visual feature associated with the prior workflow interruption; determining pixel intensities of the visual feature of the user interface associated with the second workflow state; accessing, from the memory, pixel intensities of the prior visual feature associated with the prior workflow interruption; and generating a pixel intensity score indicating a degree of similarity between the pixel intensities of the visual feature of the user interface associated with the second workflow state and pixel intensities of the prior visual feature associated with the prior workflow interruption; determining, based on the visual feature score and the pixel intensity score, that the second workflow state includes a workflow interruption; pausing the automated workflow; receiving an indication that the user input has been received; and based on receiving the indication, resuming the automated workflow.

Description

BACKGROUND Computer programs for automating a series of one or more computer tasks enable users to perform certain actions with reduced or minimal need for user input or interaction. In the process of performing these tasks, though, the program may encounter a point at which user input or interaction is required to proceed, at which point it would be advantageous if the program were able to detect this interruption and pause the automation until appropriate user input has been received. It is with respect to these and other general considerations that embodiments have been described. Also, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. SUMMARY Aspects of the present disclosure relate to detecting an interruption during an automated workflow. An automated workflow may be a series of actions performed by or with the assistance of a computer. Performing the automated workflow may involve proceeding through one or more workflow states. A workflow manager may proceed through the workflow states by interacting with one or more resources, such as a web site, a mobile application, or a database, to perform the actions associated with the workflow and to progress through the workflow states. At one or more points during the automated workflow, the workflow manager may encounter an interruption that prevents the workflow from proceeding to the next workflow state. Certain interruptions, such as a CAPTCHA, may require user input to resolve the interruption and continue the automated workflow. An interruption detection engine monitors the progress of the automated workflow from one state to the next and evaluates the workflow states to determine when an interruption has occurred. The interruption detection engine may employ a range of techniques to perform interruption detection, including without limitation techniques to examine a document object model associated with a workflow state and/or computer vision techniques to examine a user interface of a workflow state. When an interruption is detected, the workflow may be paused and the interruption presented to a user for resolution of the interruption (e.g., by providing the required user input). Upon resolution of the interruption, the workflow manager resumes the workflow and the automated workflow may progress to completion. This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS Non-limiting and non-exhaustive examples are described with reference to the following Figures. FIG. 1 illustrates an overview of an example system for an automated workflow according to aspects described herein. FIG. 2 illustrates an overview of an example method for detecting an interruption during an automated workflow. FIG. 3 illustrates an overview of an example method for detecting an interruption during an automated workflow using DOM. FIG. 4 illustrates an overview of an example method for detecting an interruption during an automated workflow using computer vision. FIG. 5 illustrates an overview of an example method for detecting an interruption during an automated workflow using a first and second detection technique. FIG. 6 is a block diagram illustrating example physical components of a computing device with which aspects of the disclosure may be practiced. FIGS. 7A and 7B are simplified block diagrams of a mobile computing device with which aspects of the present disclosure may be practiced. FIG. 8 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. FIG. 9 illustrates a tablet computing device for executing one or more aspects of the present disclosure. DETAILED DESCRIPTION In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustrations specific embodiments or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Embodiments may be practiced as methods, systems or devices. Accordingly, embodiments may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents. In examples, a computing device may perform workflow automation. As used herein, a workflow may refer to a series of one or more actions performed in a c