US-12619342-B2 - Cognitive detection of user interface errors

US12619342B2US 12619342 B2US12619342 B2US 12619342B2US-12619342-B2

Abstract

An embodiment includes detecting an interface element and an element attribute of the interface element in a series of views of a user interface, and then after an update of the user interface, detecting a candidate element and a candidate element attribute in a series of views of the updated user interface. The embodiment then determines that the updated user interface lacks any errors using a decision tree that includes comparisons of all interface elements of the user interface to corresponding candidate elements of the updated user interface. The embodiment then generates an optimized decision tree based at least in part on an analysis of the comparisons of the user interface to the updated user interface resulting in a condition that allows for the determining of a lack of errors based on comparisons of a subset of the interface elements to corresponding candidate elements.

Inventors

Yuan Feng
Yan Yan Han
Ling Zhuo
Tian Jiao Zhang
Jing Xu
Xue Ying ZHANG
SU LI HOU

Assignees

INTERNATIONAL BUSINESS MACHINES CORPORATION

Dates

Publication Date: 20260505
Application Date: 20220113

Claims (19)

1 . A computer implemented method comprising: generating, by rendering from an application executing on a data processing system, a user interface on a display apparatus, and by capturing from the rendering of the user interface a screen recording, a first sequence of image files corresponding to respective views of a first series of views of a user interface, the first sequence of image files being generated from frames of the screen recording during a first window of time; causing, by executing a detecting instruction, a detection of an interface element in the first series of views of the user interface and an element attribute of the interface element, the element attribute being descriptive of a stylistic characteristic of the interface element; updating at least one interface element of the user interface, the updating causing executing a second screen recording and automatically activating each element of the user interface during the second screen recording; generating, responsive to rendering from the application executing on the data processing system an updated user interface from the application on the display apparatus and capturing from the rendering of the updated user interface the second screen recording, a second sequence of image files corresponding to respective views of a second series of views of the updated user interface during activating each element of the user interface, the second sequence of image files being generated from frames of the second screen recording during a second window of time, the second window of time occurring after a completion of the first window of time; causing, by executing the detecting instruction, a detection of a candidate element in the second series of views of the updated user interface and a candidate element attribute of the candidate element; establishing, in a decision tree, a correspondence between a first interface element and a first candidate element responsive to a multi-modal similarity score computed using a profile of the first interface element and a profile of the first candidate element, the multi-modal similarity score indicating that the first interface element is more similar to the first candidate element relative to a second candidate element; executing, using the decision tree data structure that includes comparisons of all interface elements of the user interface to corresponding candidate elements of the updated user interface, a comparison function over all interface elements of the user interface to the corresponding elements of the updated user interface, the comparison function outputting a determination that the updated user interface lacks any stylistic errors, the stylistic errors comprising overlapping interface elements, wherein the determining that the updated interface lacks any stylistic errors comprises determining that the candidate element matches the interface element based at least in part on a comparison of the candidate element attribute to the element attribute; executing an optimization function, the optimization function generating an optimized decision tree based at least in part on a plurality of comparisons of the user interface to the updated user interface resulting in the determination that the updated user interface lacks any stylistic errors, and based on comparisons of a subset of the interface elements to corresponding candidate elements.
2 . The method of claim 1 , further comprising: generating a first data structure for an element profile of the interface element in a non-volatile computer memory, wherein the generating of the first data structure comprises mapping the element attribute to the interface element.
3 . The method of claim 2 , further comprising: detecting an element action of the interface element by activating the interface element during the first window of time and comparing views of the first series of views captured before and after the activating of the interface element.
4 . The method of claim 3 , wherein the generating of the first data structure further comprises mapping the element action to the interface element.
5 . The method of claim 4 , further comprising: detecting a candidate element action of the candidate element by activating the candidate element during the second window of time and comparing views of the second series of views captured before and after the activating of the candidate element.
6 . The method of claim 5 , further comprising: generating a second data structure for a candidate element profile of the candidate element in the non-volatile computer memory, wherein the generating of the second data structure comprises mapping the candidate element attribute and the candidate element action to the candidate element.
7 . The method of claim 1 , wherein the detecting of the element attribute comprises recognizing element text associated with the interface element in at least one of the first series of views using natural language processing.
8 . The method of claim 7 , wherein the detecting of the candidate element attribute comprises recognizing candidate text associated with the candidate element in at least one of the second series of views using the natural language processing.
9 . The method of claim 8 , wherein the comparison of the candidate element attribute to the element attribute comprises retrieving a synonym of the candidate text and comparing the synonym to the element text.
10 . The method of claim 8 , wherein the comparison of the candidate element attribute to the element attribute comprises retrieving a translation of the candidate text and comparing the translation to the element text.
11 . The method of claim 1 , wherein the comparison of the candidate element attribute to the element attribute comprises calculating a similarity score based on a plurality of aspects of the candidate element and the interface element.
12 . A computer program product, the computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising: generating, by rendering from an application executing on a data processing system, a user interface on a display apparatus, and by capturing from the rendering of the user interface a screen recording, a first sequence of image files corresponding to respective views of a first series of views of a user interface, the first sequence of image files being generated from frames of the screen recording during a first window of time; causing, by executing a detecting instruction, a detection of an interface element in the first series of views of the user interface and an element attribute of the interface element, the element attribute being descriptive of a stylistic characteristic of the interface element; updating at least one interface element of the user interface, the updating causing executing a second screen recording and automatically activating each element of the user interface during the second screen recording; generating, responsive to rendering from the application executing on the data processing system an updated user interface from the application on the display apparatus and capturing from the rendering of the updated user interface the second screen recording, a second sequence of image files corresponding to respective views of a second series of views of the updated user interface during activating each element of the user interface, the second sequence of image files being generated from frames of the second screen recording during a second window of time, the second window of time occurring after a completion of the first window of time; causing, by executing the detecting instruction, a detection of a candidate element in the second series of views of the updated user interface and a candidate element attribute of the candidate element; establishing, in a decision tree, a correspondence between a first interface element and a first candidate element responsive to a multi-modal similarity score computed using a profile of the first interface element and a profile of the first candidate element, the multi-modal similarity score indicating that the first interface element is more similar to the first candidate element relative to a second candidate element; executing, using the decision tree data structure that includes comparisons of all interface elements of the user interface to corresponding candidate elements of the updated user interface, a comparison function over all interface elements of the user interface to the corresponding elements of the updated user interface, the comparison function outputting a determination that the updated user interface lacks any stylistic errors, wherein the determining that the updated interface lacks any stylistic errors comprises determining that the candidate element matches the interface element based at least in part on a comparison of the candidate element attribute to the element attribute; executing an optimization function, the optimization function generating an optimized decision tree based at least in part on a plurality of comparisons of the user interface to the updated user interface resulting in the determination that the updated user interface lacks any stylistic errors, and based on comparisons of a subset of the interface elements to corresponding candidate elements.
13 . The computer program product of claim 12 , wherein the stored program instructions are stored in a computer readable storage device in a data processing system, and wherein the stored program instructions are transferred over a network from a remote data processing system.
14 . The computer program product of claim 12 , wherein the stored program instructions are stored in a computer readable storage device in a server data processing system, and wherein the stored program instructions are downloaded to a remote data processing system in response to a request over a network, wherein the stored program instructions are for use in a computer readable storage device associated with the remote data processing system, further comprising: program instructions to meter a use of program instructions associated with the request; and program instructions to generate an invoice based on the metered use.
15 . The computer program product of claim 12 , the operations further comprising: generating a first data structure for an element profile of the interface element in a non-volatile computer memory, wherein the generating of the first data structure comprises mapping the element attribute to the interface element.
16 . The computer program product of claim 15 , the operations further comprising: detecting an element action of the interface element by activating the interface element during the first window of time and comparing views of the first series of views captured before and after the activating of the interface element.
17 . A computer system comprising one or more processors and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the one or more processors to cause the one or more processors to perform operations comprising: generating, by rendering from an application executing on a data processing system, a user interface on a display apparatus, and by capturing from the rendering of the user interface a screen recording, a first sequence of image files corresponding to respective views of a first series of views of a user interface, the first sequence of image files being generated from frames of the screen recording during a first window of time; causing, by executing a detecting instruction, a detection of an interface element in the first series of views of the user interface and an element attribute of the interface element, the element attribute being descriptive of a stylistic characteristic of the interface element; updating at least one interface element of the user interface, the updating causing executing a second screen recording and automatically activating each element of the user interface during the second screen recording; generating, responsive to rendering from the application executing on the data processing system an updated user interface from the application on the display apparatus and capturing from the rendering of the updated user interface the second screen recording, a second sequence of image files corresponding to respective views of a second series of views of the updated user interface during activating each element of the user interface, the second sequence of image files being generated from frames of the second screen recording during a second window of time, the second window of time occurring after a completion of the first window of time; causing, by executing the detecting instruction, a detection of a candidate element in the second series of views of the updated user interface and a candidate element attribute of the candidate element; establishing, in a decision tree, a correspondence between a first interface element and a first candidate element responsive to a multi-modal similarity score computed using a profile of the first interface element and a profile of the first candidate element, the multi-modal similarity score indicating that the first interface element is more similar to the first candidate element relative to a second candidate element; executing, using the decision tree data structure that includes comparisons of all interface elements of the user interface to corresponding candidate elements of the updated user interface, a comparison function over all interface elements of the user interface to the corresponding elements of the updated user interface, the comparison function outputting a determination that the updated user interface lacks any stylistic errors, wherein the determining that the updated interface lacks any stylistic errors comprises determining that the candidate element matches the interface element based at least in part on a comparison of the candidate element attribute to the element attribute; executing an optimization function, the optimization function generating an optimized decision tree based at least in part on a plurality of comparisons of the user interface to the updated user interface resulting in the determination that the updated user interface lacks any stylistic errors, and based on comparisons of a subset of the interface elements to corresponding candidate elements.
18 . The computer system of claim 17 , the operations further comprising: generating a first data structure for an element profile of the interface element in a non-volatile computer memory, wherein the generating of the first data structure comprises mapping the element attribute to the interface element.
19 . The computer system of claim 18 , the operations further comprising: detecting an element action of the interface element by activating the interface element during the first window of time and comparing views of the first series of views captured before and after the activating of the interface element.

Description

BACKGROUND The present invention relates generally to a method, system, and computer program product for data processing. More particularly, the present invention relates to a method, system, and computer program product for cognitive detection of user interface errors. Many software applications include a graphical user interface (also referred to herein more simply as a user interface) that includes various interface elements. The interface elements are computer-generated elements rendered on a display that provide input and/or output functionality for users. Examples of typical interface elements include input controls (e.g., checkboxes, buttons, and text fields), navigational controls (e.g., search fields, sliders, and icons), informational components (e.g., progress bars, message boxes, and tooltips), and containers (e.g., accordions). While the software is being developed or updated, the user interface may be frequently tested to detect any unforeseen errors that may occur because of changes to the source code. For example, when a software application is created according to test-driven development practices, test cases for the user interface are created and repeatedly performed as the software is being developed to validate added functionality and confirm continued validation of previously-implemented functionality. The test cases are typically based on software requirements and successful completion of each test confirms satisfaction of the corresponding requirement. SUMMARY The illustrative embodiments provide for cognitive detection of user interface errors. An embodiment includes generating a first sequence of image files corresponding to respective views of a first series of views of a user interface for a software application under development, the first sequence of image files being generated during a first window of time. The embodiment also includes detecting an interface element in the first series of views of the user interface and an element attribute of the interface element, the element attribute being descriptive of the interface element. The embodiment also includes generating, after an update of the user interface resulting in an updated user interface, a second sequence of image files corresponding to respective views of a second series of views of the updated user interface, the second sequence of image files being generated during a second window of time, the second window of time occurring after completion of the first window of time. The embodiment also includes detecting a candidate element in the second series of views of the updated user interface and a candidate element attribute of the candidate element. The embodiment also includes determining that the updated user interface lacks any errors using a decision tree that includes comparisons of all interface elements of the user interface to corresponding candidate elements of the updated user interface, where the comparisons comprise determining that the candidate element matches the interface element based at least in part on a comparison of the candidate element attribute to the element attribute. The embodiment also includes generating an optimized decision tree based at least in part on an analysis of the comparisons of the user interface to the updated user interface resulting in a condition that allows for the determining of a lack of errors based on comparisons of a subset of the interface elements to corresponding candidate elements. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the embodiment. An embodiment includes a computer usable program product. The computer usable program product includes a computer-readable storage medium, and program instructions stored on the storage medium. An embodiment includes a computer system. The computer system includes a processor, a computer-readable memory, and a computer-readable storage medium, and program instructions stored on the storage medium for execution by the processor via the memory. BRIEF DESCRIPTION OF THE DRAWINGS The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of the illustrative embodiments when read in conjunction with the accompanying drawings, wherein: FIG. 1 depicts a cloud computing environment according to an embodiment of the present invention; FIG. 2 depicts abstraction model layers according to an embodiment of the present invention; FIG. 3 depicts a block diagram of an example service infrastructure that includes a user interface (UI) testing system in accordance with an illustrative embodiment; FIG. 4 depicts a block diagram of an example UI testing system in accordance wi