Search

CN-121986312-A - Remote eye gaze cursor control technique

CN121986312ACN 121986312 ACN121986312 ACN 121986312ACN-121986312-A

Abstract

A system adapted to interact with a user by monitoring the activity of the user's eye gaze, the system comprising a user webcam configured to determine the position of the user's eye gaze on a user display configured to display digital content to the user, an operator display configured to view the live interaction of the user's eye gaze from the movement of the eye gaze controlled cursor projected in the operator display, a processor with a gaze estimation module for receiving eye movement and head movement signals from the user webcam as calibration data for estimating the eye gaze, the processor configured to generate an interaction stimulus to the user display, interact with the interaction stimulus via a user gaze based interaction or a blink based interaction, and analyze and determine a user focus based on eye tracking of interactions from the interaction stimulus.

Inventors

  • J. Aaron
  • G. L.T. Ralph

Assignees

  • 鱿鱼目私人有限公司

Dates

Publication Date
20260505
Application Date
20240806
Priority Date
20230808

Claims (20)

  1. 1. A system adapted to interact with a user by monitoring activity of eye gaze of the user, the system comprising: A user network camera configured to determine a location of a user's eye gaze on a user display configured to display digital content to the user; An operator display configured to view the live interaction of the user's eye gaze from movement of the eye gaze controlled cursor projected in the operator display; A processor having a gaze estimation module configured to create a calibration session for the user to join, wherein upon joining, the gaze estimation module receives eye movement and head movement signals from the user webcam as calibration data for estimating eye gaze; estimating a user gaze direction and focus to predict a location at which the user is gazing on the display, wherein the predicted user eye gaze is represented by movements of the eye gaze controlled cursor in the operator display and the user display; Creating a test session for the operator and the user to test eye gaze accuracy, and Creating a master session for the operator and the user to join; generating an interactive stimulus to the user display; Interacting with the interaction stimulus via a user gaze-based interaction or a blink-based interaction, wherein a predefined period of gaze-based interaction or blink-based interaction allows the eye gaze controlled cursor to execute on the interaction stimulus; the user focus is analyzed and determined based on eye tracking of interactions from the interaction stimulus.
  2. 2. The system of claim 1, wherein the processor provides user eye-tracking accuracy calibration data to the operator display and the user display.
  3. 3. The system of any of claims 1-2, calibrating user eye tracking until the calibration data passes above an accuracy threshold, wherein the accuracy threshold is greater than 50% for passing calibration.
  4. 4. A system according to claim 3, wherein the accuracy threshold is higher than 75% for passing calibration.
  5. 5. The system of any of claims 3 to 4, wherein the calibration session allows the operator and the user access to repeat the calibration.
  6. 6. The system of any of claims 3 to 4, wherein when passing calibration, the processor sends calibration data to the operator display whether to recalibrate or initiate the test session or the master session.
  7. 7. The system of any of claims 1-6, wherein the processor further comprises a cursor tracking engine, wherein the cursor tracking engine begins tracking cursor control when permission to access the user cursor control is received from the host, and wherein the system continues to passively calculate user eye gaze when the host permission is not received.
  8. 8. The system of any of claims 1-7, wherein the user display has a first set of interface controls and wherein the operator display has a second set of interface controls, wherein the second set of interface controls is greater in number than the first set of interface controls and wherein the second set of interface controls consists of the first set of interface controls, and wherein the second set of interface controls consists of the first set of interface controls such that the operator can control the user's display.
  9. 9. The system of claim 8, wherein the user display has a first set of buttons including a user video live stream display button, an end of the virtual telemedicine session button, an open or close user video recording button, a screen capture button, an open or close microphone audio button.
  10. 10. The system of any of claims 8 to 9, wherein the operator display has a second set of buttons including a user video live stream display button, an end of the virtual telemedicine session button, an open or close user video record button, a screen grab button, an open or close microphone audio button, a calibrate button, a shared screen button, a user tracking eye button.
  11. 11. The system of claim 10, wherein the corresponding user button feature starts, stops, or updates when the operator selects a button.
  12. 12. The system of any of claims 1 to 11, wherein the calibration session is broadcast, and wherein the test session is broadcast, and wherein the master session is broadcast.
  13. 13. The system of any of claims 1 to 12, wherein a second user joins the calibration session or the test session or the main session, wherein the predicted second user eye gaze is represented by movement of a cursor controlled by the operator display and a second eye gaze in the user display.
  14. 14. The system of claim 13, wherein the second eye-gaze controlled cursor for the second user is different from the eye-gaze controlled cursor for the first user.
  15. 15. The system of any one of claims 1 to 14, wherein the user is one selected from the group consisting of a normal user, a clinician, a user with disability, a user with neurological disability, and wherein the operator is one selected from the group consisting of a normal user, a clinician, a user with disability, and a user with neurological disability.
  16. 16. The system of any one of claims 1 to 15, wherein the digital content is at least one selected from the group consisting of interactive content and static content.
  17. 17. The system of any one of claims 1 to 16, wherein the system is embedded within a digital communication tool.
  18. 18. The system of claim 17, wherein the digital communication tool is one selected from the group consisting of a web app, a virtual conference platform, a mobile application, an application for communication and interaction in digital format.
  19. 19. The system of any of claims 1 to 18, wherein the processor is configured to send an alert to the operator display and the user display when the user focus falls below a predetermined attention threshold.
  20. 20. The system of claim 19, wherein the alert is at least one selected from the group consisting of a visual alert, an audio alert, and a text alert.

Description

Remote eye gaze cursor control technique Technical Field The present invention or disclosure relates generally to eye tracking, and more particularly to systems and methods for facilitating remote eye tracking control for use as part of various audio-visual virtual conference platforms including for telemedicine. Background Individuals with neurological disabilities may have limited control over movement and speech and rely solely on their eye gaze for communication. As COVID-19 epidemic situations transfer face-to-face activities such as rehabilitation courses and the like to the internet, the learning quality and health and social care results of the people are reduced. Because of the inability to obtain technological advances, including special tools that help with computer and telemedicine experiences, clinicians who cooperate with these people face challenges in interacting and participating during online conferences and providing adequate support for face-to-face conferences. The application of information communication technology in the rehabilitation field is called tele-rehabilitation. This telecommunications-based practice can provide remote support, assessment and information to individuals, including those with neurological disabilities, without requiring them to leave the home. Previous reports indicate that remote rehabilitation improves the well-being of a neurological disability person because it can increase the chances of getting rehabilitation services and healthcare professionals. This effect is particularly pronounced in areas of insufficient service and limited access to these auxiliary resources. Thus, the challenges faced are twofold 1) the opportunity to provide health and social care through remote rehabilitation, and 2) the ability of the tool to effectively support individuals, including patients with neurological disabilities. Current technological advances have adapted the use of eye gaze control to allow individuals with disabilities with speech disorders to gain more health and social care. Many eye gaze devices are off the shelf, including Tobii Dynavox portable eye trackers and eye gaze tablets. Research shows that the special eye sight tracking devices enhance the participation of children with speech disorder and nerve disability and improve early communication skills. However, they are not accessible due to their high cost and inability to integrate with teleconferencing platforms. Furthermore, clinicians and educators currently have limited opportunities to remotely monitor and evaluate eye gaze as a communication aid. The invention described herein seeks to address this gap in remote eye gaze control technology in teleconferencing platforms, as well as improve communication, learning and support between clinicians, researchers and educators and the neurological disabled individual, which may be used when a rehabilitation session is not possible in the field. It is worth mentioning that the invention is not limited, nor is it conditioned for use by only individuals with neurological disabilities, other persons may use it for the same or similar benefits. Any discussion of the prior art throughout the specification should in no way be considered as an admission that such prior art is widely known or forms part of common general knowledge in the field. Disclosure of Invention Problems to be solved The present invention relates to a remote eye gaze cursor control system embedded within a digital communication tool, such as a web app, or virtual conference platform, or mobile application, or application for communication and interaction in digital format. The system includes creating a virtual session or broadcast session in which a moderator or operator and a participant or user can join and broadcast remote estimates of the participant's eye gaze. The application may be used by individuals with neural diversity for use during an online telemedicine conference. The present invention may include 3 main modules, (1) API connections to which WebSocket or web app is installed, (2) face and eye detection, and (3) cursor control estimation, which will be described herein. For simplicity, each of these modules is summarized in the following three paragraphs. For the purposes of this report, the terms clinician, moderator, operator, and author are used interchangeably, with the terms participant, student, and user being the same. The present invention may be packaged as a stand-alone web app on a server that accompanies an audiovisual virtual conferencing platform, or embedded as an application or tool that may be deployed within a teleconferencing platform such as Coviu. The intent of the two systematic deviations may be the same, so that both the clinician and the user can view the communication (by virtual means) while also being able to communicate through the invention as a live broadcast view with the user's eye gaze onto the main conversation window. At a high level, both s