US-12627897-B2 - Automatic video recording alignment

Abstract

Methods and apparatus for dynamic video recording alignment are provided. A video alignment system receives a first signal indicating that an interaction device comprising a screen is in use. Responsive to the first signal, the system instructs the interaction device to record a series of images displayed on the screen. The system detects flicker in a video recording generated by a camera and depicting the screen, where the video recording comprises one or more defective frames contributing to the flicker. Upon detecting the flicker, the system retrieves one or more images, from the series of images, where the one or more images are recorded by the interaction device at moments of corresponding to the flicker. The system superimposes the one or more images onto the one or more defective frames in the video recording.

Inventors

• Yihsin HUANG
• Wei-Yi Hsuan
• Wen-Fu Tsai
• Yin-Chen CHOU

Assignees

• TOSHIBA GLOBAL COMMERCE SOLUTIONS, INC.

Dates

Publication Date

20260512

Application Date

20240410

Claims (17)

1. 1 . A method comprising: receiving a first signal indicating that an interaction device comprising a screen is in use; responsive to the first signal, instructing the interaction device to record a series of images displayed on the screen; detecting flicker in a video recording generated by a camera and depicting the screen, wherein the video recording comprises one or more defective frames contributing to the flicker; upon detecting the flicker, retrieving one or more images, from the series of images, wherein the one or more images are recorded by the interaction device at moments of corresponding to the flicker; receiving a second signal indicating an end of use of the interaction device; responsive to the second signal, instructing the interaction device to stop recording the series of images displayed on the screen; and superimposing the one or more images onto the one or more defective frames in the video recording.
2. 2 . The method of claim 1 , wherein the flicker results in the one or more images displayed on the screen not visible in the video recording.
3. 3 . The method of claim 1 , wherein the flicker is caused by a discrepancy between a refresh rate of the screen and a frame rate of the camera.
4. 4 . The method of claim 1 , wherein the interaction device comprises at least one of a self-checkout station or an interactive kiosk.
5. 5 . The method of claim 1 , further comprising: receiving a refresh rate of the screen from the interaction device; receiving a frame rate from the camera; comparing the refresh rate and the frame rate within a defined time interval; and upon detecting that the refresh rate is different from the frame rate, initiating a flicker examination on the video recording received from the camera.
6. 6 . The method of claim 1 , wherein the first and second signals are generated based on motion detection by the camera.
7. 7 . A system, comprising: one or more processors; one or more memories storing a program, which, when executed on any combination of the one or more processors, performs operations, the operations comprising: receiving a first signal indicating that an interaction device comprising a screen is in use; responsive to the first signal, instructing the interaction device to record a series of images displayed on the screen; detecting flicker in a video recording generated by a camera and depicting the screen, wherein the video recording comprises one or more defective frames contributing to the flicker; upon detecting the flicker, retrieving one or more images, from the series of images, wherein the one or more images are recorded by the interaction device at moments of corresponding to the flicker; receiving a second signal indicating an end of use of the interaction device; responsive to the second signal, instructing the interaction device to stop recording the series of images displayed on the screen; and superimposing the one or more images onto the one or more defective frames in the video recording.
8. 8 . The system of claim 7 , wherein the flicker results in the one or more images displayed on the screen not visible in the video recording.
9. 9 . The system of claim 7 , wherein the flicker is caused by a discrepancy between a refresh rate of the screen and a frame rate of the camera.
10. 10 . The system of claim 7 , wherein the interaction device comprises at least one of a self-checkout station or an interactive kiosk.
11. 11 . The system of claim 7 , wherein the program, which, when executed on any combination of the one or more processors, performs the operations further comprising: receiving a refresh rate of the screen from the interaction device; receiving a frame rate from the camera; comparing the refresh rate and the frame rate within a defined time interval; and upon detecting that the refresh rate is different from the frame rate, initiating a flicker examination on the video recording received from the camera.
12. 12 . The system of claim 7 , wherein the first and second signals are generated based on motion detection by the camera.
13. 13 . One or more non-transitory computer-readable media containing, in any combination, computer program code that, when executed by operation of a computer system, performs operations comprising: receiving a first signal indicating that an interaction device comprising a screen is in use; responsive to the first signal, instructing the interaction device to record a series of images displayed on the screen; detecting flicker in a video recording generated by a camera and depicting the screen, wherein the video recording comprises one or more defective frames contributing to the flicker; upon detecting the flicker, retrieving one or more images, from the series of images, wherein the one or more images are recorded by the interaction device at moments of corresponding to the flicker; receiving a second signal indicating an end of use of the interaction device; responsive to the second signal, instructing the interaction device to stop recording the series of images displayed on the screen; and superimposing the one or more images onto the one or more defective frames in the video recording.
14. 14 . The one or more non-transitory computer-readable media of claim 13 , wherein the flicker results in the one or more images displayed on the screen not visible in the video recording.
15. 15 . The one or more non-transitory computer-readable media of claim 13 , wherein the flicker is caused by a discrepancy between a refresh rate of the screen and a frame rate of the camera.
16. 16 . The one or more non-transitory computer-readable media of claim 13 , wherein the interaction device comprises at least one of a self-checkout station or an interactive kiosk.
17. 17 . The one or more non-transitory computer-readable media of claim 13 , wherein the first and second signals are generated based on motion detection by the camera.

Description

BACKGROUND Understanding user behavior, preferences, and interactions with a variety of computing systems is useful for improving the computing systems themselves, as well as to help improve operational efficiency and enhancing the user's overall experience. One method to achieve this understanding is by analyzing video footage, which captures how users utilize the computing systems, including their navigation patterns, product or selection preferences, and challenges or difficulties they met during the interaction process. The effectiveness of the video analysis largely depends on the quality of the captured footage. High-quality video footage allows for more accurate observations and analyses of user behavior. However, discrepancies between the camera's frame rate and the computing system monitor's refresh rate can lead to issues such as flashing or flickering in the recorded footage. The resulting flicker or flashing may cause video quality distortion and obscure details of the interaction, making it challenging to conduct an accurate analysis of the video footage. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 depicts an example environment for monitoring user interactions with computing systems, according to some embodiments of the present disclosure. FIG. 2 depicts an example video player interface displaying video footage captured by in-store cameras, according to some embodiments of the present disclosure. FIG. 3 depicts an example video alignment system, according to some embodiments of the present disclosure. FIG. 4 depicts an example method for the dynamic superimposition of recorded screen images onto flicker-affected frames of video recordings, according to some embodiments of the present disclosure. FIG. 5 depicts an example method for dynamic user interaction detection and screen image recording, according to some embodiments of the present disclosure. FIG. 6 is a flow diagram depicting an example method for dynamic video recording alignment, according to some embodiments of the present disclosure. FIG. 7 depicts an example computing device configured to perform various aspects of the present disclosure, according to some embodiments of the present disclosure. DETAILED DESCRIPTION In at least one example, the present disclosure relates to automatic video recording alignment. In some embodiments, the present disclosure provides a method and system that addresses flicker in video recordings caused by discrepancies between a monitor's refresh rate and a camera's frame rate. An enterprise site (such as a retail store) usually has cameras installed around self-checkout areas to capture customers' behavior and interactions with self-checkout devices. A detailed analysis of this footage can provide various information, including customers' shopping preferences, the efficiency and/or effectiveness of the self-checkout system, and any challenges customers encounter during their interactions with the self-checkout devices. Furthermore, the analysis may reveal trends in customer behavior, such as peak usage periods for self-checkout devices, and/or preferences for interactive features on the devices. By utilizing the data, business owners may further optimize the self-checkout system, such as by adjusting the current menu layout or settings to better meet customer preferences, enhancing user-friendly features, replacing frequently used features for easier access, and introducing clearer on-screen instructions. These adjustments are configured to streamline the self-checkout process, enhance the store's overall operational efficiency, and/or improve the customer shopping experience. Although some aspects of the present disclosure discuss self-checkout systems in retail environments, aspects of the present disclosure are readily applicable to a wide variety of computing systems where users interact with graphical user interfaces (GUIs). For effective data analysis, high-quality video recording (or footage) is often useful, such as for accurately capturing the content of the monitor or touchscreen interface of the self-checkout device that the customer is interacting with. However, a challenge arises due to the discrepancy between the frame rate of the video recording and the refresh rate of the monitor or touchscreen display. This discrepancy often results in flickering in the recorded video, which can severely impact the clarity and usability of the video for data analysis. The flickering makes it difficult to identify the screen content that the customer is interacting with, and therefore compromises the system's capability to accurately analyze and interpret customer behavior. The present disclosure introduces a video alignment system configured to mitigate the effects of flicker in video recordings. The system achieves this by automatically superimposing clear checkout images (e.g., screenshots), captured by interactives devices, over the monitor of the checkout system as depicted within frames affected by