Search

WO-2026096960-A1 - SYSTEMS AND METHODS FOR USING A MACHINE LEARNING MODEL AND HEAD MOUNTED DEVICE FOR MEDICAL PROCEDURES

WO2026096960A1WO 2026096960 A1WO2026096960 A1WO 2026096960A1WO-2026096960-A1

Abstract

In an embodiment, a processing system receives sensor data from a head-mounted device (HMD) during a medical procedure. The processing system determines a gaze direction of a wearer of the HMD using the sensor data. The processing system determines that the gaze direction is directed to a content of a plurality of content displayed for the medical procedure. The processing system determines context for the medical procedure based on the content. The processing system generates, using a trained machine learning model taking the context as input, an annotation for a record of the medical procedure.

Inventors

  • RAZZAQUE, SHARIF
  • SILVA, Jonathan, R.
  • SILVA, Jennifer, N. Avari
  • TAS, Berk
  • SOUTHWORTH, Michael, K.

Assignees

  • SentiAR, Inc.

Dates

Publication Date
20260507
Application Date
20251031
Priority Date
20241031

Claims (15)

  1. 1. A method comprising: receiving sensor data from a head-mounted device (HMD) during a medical procedure; determining a gaze direction of a wearer of the HMD using the sensor data; determining that the gaze direction is directed to a content of a plurality of content displayed for the medical procedure; determining context for the medical procedure based on the content; and generating, using a trained machine learning model taking the context as input, an annotation for a record of the medical procedure.
  2. 2. The method of claim 1, further comprising: receiving an input from the wearer of the HMD using additional sensor data from the HMD; generating a manual annotation using the input; and further training the trained machine learning model using the manual annotation.
  3. 3. The method of claim 2, further comprising: determining one or more words spoken by the wearer of the HMD by processing the input; and responsive to determining that the one or more words are associated with the medical procedure, generating the manual annotation to include the one or more words.
  4. 4. The method of any of claims 2-3, further comprising: determining a ranking of a plurality of annotations including at least the manual annotation and the annotation generated using the trained machine learning model, wherein the record of the medical procedure reflects the ranking.
  5. 5. The method of claim 4, wherein the record of the medical procedure is a timeline including graphical representations of the manual annotation and the annotation generated using the trained machine learning model, and wherein the graphical representations are displayed with emphasis to reflect the ranking.
  6. 6. The method of any of claims 4 or 5, wherein the plurality of annotations further includes an additional annotation generated using input from another user different than the wearer of the HMD.
  7. 7. The method of any of claims 1-6, further comprising: determining one or more words spoken by the wearer of the HMD by processing the input; and Atty Docket No.: 34237-64696/WO responsive to determining that the one or more words are associated with the medical procedure, generating a manual annotation to include the one or more words.
  8. 8. The method of any of claims 1-7, further comprising: providing a virtual graphic for the medical procedure for display by the HMD; identifying 2D content for the medical procedure displayed by a device separate from the HMD; and wherein determining that the gaze direction is directed to the content of the plurality of content displayed for the medical procedure comprises determining that the gaze direction is directed to the virtual graphic or the 2D content.
  9. 9. The method of any of claims 1-8, further comprising: requesting, from the wearer of the HMD, confirmation of the annotation generated using the trained machine learning model; and responsive to receiving the confirmation from the wearer of the HMD, including the annotation in the record of the medical procedure.
  10. 10. The method of any of claims 1-9, further comprising: receiving, from the wearer of the HMD, a modification of the annotation generated using the trained machine learning model; and further training the trained machine learning model using the modification.
  11. 11. The method of any of claims 1-10, further comprising: providing electrogram data for the medical procedure for display by the HMD; determining a measurement of the electrogram data; and wherein the trained machine learning model generates the annotation by taking the measurement as input.
  12. 12. The method of any of claims 1-11, further comprising: determining that the gaze direction is directed to a virtual content displayed by the HMD; responsive to determining that the virtual content is at least partially obstructed from a point of view of the wearer of the HMD, modifying a position of the virtual content; and further training the trained machine learning model using the modified position of the virtual content.
  13. 13. The method of any of claims 1-10, wherein the trained machine learning model is trained using previous inputs from the wearer of the HMD and from other medical procedures. Atty Docket No.: 34237-64696/WO
  14. 14. A non-transitory computer-readable storage medium storing instructions that when executed by one or more processors cause the one or more processors to perform steps of any of the methods of claims 1-13.
  15. 15. A system comprising: a head-mounted device (HMD); and a non-transitory computer-readable storage medium storing instructions that when executed by one or more processors cause the one or more processors to perform steps of any of the methods of claims 1-13.

Description

Atty Docket No.: 34237-64696/WO SYSTEMS AND METHODS FOR USING A MACHINE LEARNING MODEL AND HEAD MOUNTED DEVICE FOR MEDICAL PROCEDURES CROSS REFENCE TO RELATED APPLICATION [0001 ] This application claims the benefit of priority to U.S. Provisional Application No. 63/714,697, filed on October 31, 2024, which is incorporated herein by reference in its entirety for all purposes. TECHNICAL FIELD [0002] This disclosure generally relates to training machine learning models within a medical computing environment. BACKGROUND [0003] Conventional methods of summarizing a medical procedure, such as an Electrophysiology mapping and ablation procedure, are time consuming and may not capture enough relevant detail for medical or billing purposes. [0004J There may also be circumstances (which are currently unknown or routinely missed) during the medical procedure that are related to opportunities to improve patient outcomes (for example, missed communication events, staff unknowingly compromising the sterile field leading to an increased risk of infection, distracting unrelated conversion causing staff to miss clinically important events) or increase efficiency (for example, an missed request to record an event, or the position of a certain piece of large equipment causing staff to have to frequently wait for each other to walk around it, causing delays in the procedure). The results of such medical-procedure analysis may also be useful for training staff and physicians. [0005J Medical procedures are performed with the aid of imaging and other systems that present data on 2D display screens (e.g., ultrasound, fluoroscopy, endoscopy, navigation, electroanatomic mapping, electrogram recording systems, electronic medical record systems, etc.). Thus, physicians and clinical staff are often looking at a variety of 2D displays and real -world objects (e.g., the patient and procedure site, the physicians’ hands, medical instruments, other people in the procedure room, etc.) while performing the medical procedure. [0006] During many medical procedures, the physician is aided by staff members (assistants). Some of these assistants operate computer-based systems (e.g., electronic medical record systems, ultrasound scanners, electrogram signal recording systems, surgical Atty Docket No.: 34237-64696/WO navigation systems, medical-image viewers, cardiac stimulators, etc.) on behalf of the physician, whose hands are sterile and/or occupied. By partially or fully automating the staff’s computer-based tasks, the clinic may reduce staff workload, thereby reducing costs, time, and error rates. [0007] Conventional artificial-intelligence / machine-learning in clinical use for tasks such as transcription and report generation are susceptible to “hallucinate” and generate transcription text or report text that does not faithfully represent things that were actually said or done or happened during the medical procedure. A human reviewing the transcript or report may miss these hallucination artifacts and thus not remove or correct them in the submitted medical report. SUMMARY [0008] In an embodiment, a method comprises receiving sensor data from a headmounted device (HMD) during a medical procedure. The method further comprises determining a gaze direction of a wearer of the HMD using the sensor data. The method further comprises determining that the gaze direction is directed to a content of a plurality of content displayed for the medical procedure. The method further comprises determining context for the medical procedure based on the content. The method further comprises generating, using a trained machine learning model (e.g., having the structure described with respect to FIG. 7) taking the context as input, an annotation for a record of the medical procedure. [0009] In an embodiment, the method further comprises receiving an input from the wearer of the HMD using additional sensor data from the HMD; generating a manual annotation using the input; and further training the trained machine learning model using the manual annotation. [00010] In an embodiment, the method further comprises determining one or more words spoken by the wearer of the HMD by processing the input; and responsive to determining that the one or more words are associated with the medical procedure, generating the manual annotation to include the one or more words. [0001.1] In an embodiment, the method further comprises determining a ranking of a plurality of annotations including at least the manual annotation and the annotation generated using the trained machine learning model, wherein the record of the medical procedure reflects the ranking. In an embodiment, the record of the medical procedure is a timeline including graphical representations of the manual annotation and the annotation generated Atty Docket No.: 34237-64696/WO using the trained machine learning model, and wherein the graphical representations are displayed with emphasis to reflect the ranking.