Search

EP-4207713-B1 - METHOD AND SYSTEM FOR EVENT BASED, PRIVACY AWARE CAPTURING OF PARTIAL SCREEN CHANGES FOR DEVICES WITH RESTRICTED RESOURCES

EP4207713B1EP 4207713 B1EP4207713 B1EP 4207713B1EP-4207713-B1

Inventors

  • Pereiro Parejo, Delfin
  • Criado, Lluis
  • MUNOZ, CARLOS

Dates

Publication Date
20260506
Application Date
20230102

Claims (14)

  1. A computer-implemented method for reconstructing user interactions with an application executing on a mobile device (100), comprising: - receiving, by an agent (102), an event notification (109) for an event (245) from an operating system of the mobile device (100) observed by an application (101), where the event (245) is indicative of a touch on a display (105) and the event notification (109) includes an identifier (231) for the application (101) and a particular location on the display (105) at which the touch occurred, such that the agent (102) is instrumented in the application; - mapping, by the agent (102), the particular location on the display (105) to a particular graphical user interface, GUI, element using GUI structure data from the application (101), where the particular GUI element aligns with the touch and the GUI structure data defines layout and location (253) of GUI elements displayable by the application (101); - determining, by the agent (102), a location on the display to capture a screenshot of the particular GUI element using the layout and location (253) of the particular GUI element from the GUI structure data; - capturing, by the agent (102), a screenshot of the particular GUI element, where the screenshot is a portion of the display containing the particular GUI element; - creating, by the agent (102), session data for user interactions with the application (101), where the session data includes the identifier (231) for the application (101), the location (253) of the particular GUI element, and the screenshot; - storing, by the agent (102), the session data in memory of the mobile device (100); and - reconstructing user interactions with the application (101) using the session data.
  2. The method of claim 1 further comprises - detecting, by the agent (102), a triggering condition occurring on the mobile device (100); - adding, by the agent (102), data for the trigger condition to the session data; - sending, by the agent (102), the session data over a computer network (130) to a monitoring server (140), where the session data is sent in response to detecting the triggering condition; and - reconstructing the user interactions and the trigger condition at the monitoring server (140).
  3. The method of any of the preceding claims wherein creating session data further comprises creating an event record (121, 1107) and a screenshot record (122) for each event (245), where each event record (121, 1107) includes the identifier (231) for the application (101), an event type (221, 251), timing data for the event (245), position of the affected display element, size of the affected display element and a reference (257) to a corresponding screenshot record, and the screenshot record (122) includes the screen content data.
  4. The method of claim 3 wherein the reference (257) to a corresponding screenshot record is defined as a hash value from a hash function applied to the screen content data.
  5. The method of any of the preceding claims wherein creating session data further comprises determining visibility of the affected display element and storing the session data only when the affected display element is visible on the display (105).
  6. The method of claim 5 wherein determining visibility of the affected display element further includes checking visibility status of display elements having an ancestral relationship with the affected display element.
  7. The method of any of the preceding claims wherein creating session data further comprises determining whether the affected display element contains private information and excluding the affected display elements containing private information from the session data.
  8. The method of any one of the claims 1 to 6 wherein creating session data further comprises determining whether the affected display element contains private information and storing the session data with a tag when the affected display element contains private information, where the tag indicates that the affected display element contains private information.
  9. The method of claim 8 wherein determining whether the affected display element contains private information further includes checking whether display elements having an ancestral relationship with the affected display element contain private information.
  10. The method of claim 8 or 9 further comprises reconstructing user interactions with the application (101) by obscuring the screen content data when the tag indicates that the affected display element contains private information.
  11. The method of claim 2 wherein the triggering condition is further defined as an undesired operating condition on the mobile device (100).
  12. The method of claim 4 wherein creating the screenshot record (122) for a given event (245) further comprises using the hash value from the screen content data to determine whether a screenshot record (122) with identical hash value is already stored on the mobile device (100); and, in response to a screenshot record (122) with identical hash value is stored on the device, setting the reference (257) to the corresponding screenshot record of the event record (121, 1107) to refer to the already stores screenshot record (122).
  13. The method of claim 10, where obscuring the screen content data further comprises suppress capturing of screen content data and, during reconstructing user actions, using position and size of the affected display element to create masking content data.
  14. A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to perform the method of any of the preceding claims.

Description

The invention generally relates to the field of reconstructing user interactions with an application executing on a mobile device. Background Native mobile applications gained importance in the last years and became an important interaction channel with end-users and they also became an important revenue channel. This raised the importance of monitoring capabilities for this type of application. Knowledge of real-time functional and performance status data for individual instances of native mobile applications, as operated on mobile devices of end-users is inevitable to judge the operation status of those applications and to identify malfunctions and to execute appropriate countermeasures if required. Next to this standard monitoring requirements, also insight into interaction patterns of end-users with monitored applications is required. The desired level of visibility includes monitoring data for the replay or simulation of observed user interactions, including the reconstruction of the user interface of the monitored application as it was perceived by the end-user. This user interaction replay, or session recording data may be used to identify common, frequent user behavior, which may in turn be used to optimize the user-interface of the monitored applications. In addition, this data may be used to identify the cause of observed undesired application behavior, like unexpected performance degradations or application crashes. Especially user session data may be of interest that describes the user interaction sequence that happened before those undesired behavior occurred. This captured user interaction data may then be analyzed to identify user activities that may have led to the undesired behavior. Privacy protection rules, like the General Data Protection Regulations (GDPR), require monitoring systems that also observe user interaction data that may contain sensitive, private data, to identify such sensitive private data and to remove it from monitoring data before it is stored, processed, analyzed, or viewed. Mobile applications are typically executed on mobile devices which have limited computing, storage, and communication resources. Additional requirements for monitoring systems arise from those circumstances, like a small memory footprint with a guaranteed maximal storage space on the device or minimized network communication activity that is adapted to and optimized for the network connection currently used by the mobile device. State of the art user interaction systems for mobile devices use video capturing systems for user interactions that capture screen shots with a fixed capture rate and perform on-the-fly compression of captured monitoring data. Although those conventional approaches are easy to implement and to deploy, as they typically do not require knowledge or modifications of the interna of the monitored applications, they show some considerable shortcomings. First, as capturing is performed at a fixed rate, the created monitoring data may show large sequences of inactivity, and as capturing is performed at a fixed rate which is independent of activities related to the monitored application, monitoring data may not contain important user interaction activity that happened between screen captures. Second, the on-the-fly compression creates a storage form in which individual user interaction data is not identifiable without decompression and replay of the created movie. This makes it extremely difficult to identify and mask private user data properly. Given those shortcomings there is demand in the field for a user interaction monitoring system that matches the requirements of mobile environments or environments with limited resources and that also supports efficient protection of private end-user data. US 2019/1715421 A1 discloses a system and method for the real-user capable monitoring of the rendering process of dynamically updated web pages. The proposed approach deploys a browser agent to content loaded and displayed by a web browser which monitors resource load activities caused by the processing of the content by the browser and in-place content modifications cause by the executions of scripts and XHR communication by the browser. Those resource load and content modification activities affecting the visible area of the browser are identified and a point in time is calculated at which the visualization of the content for the visible area is complete. Further performance monitoring data describing the performance of the rendering process depending on the percentage of finished portions of the visible area over time is calculated. In addition, data describing the rendering process in detail may be captured by the browser agent which may be used for a later playback of the rendering process. US 2017/019489 A1 discloses methods for recording a session of user interaction with a website for subsequent replay. The methods comprises the steps of receiving in the memory a webpage from the w