EP-4736183-A1 - SYSTEM AND METHOD FOR DETECTING AND REMOVING NON-SURGICAL DATA

EP4736183A1EP 4736183 A1EP4736183 A1EP 4736183A1EP-4736183-A1

Abstract

Systems and methods to detect and remove non-surgical information include receiving a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receiving an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determining, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream, and modifying the data stream in response to determining that the non-surgical information is present in the data stream.

Inventors

PERREAULT, CONOR
LIU, XI
JARC, ANTHONY M.
GUO, RUI
WANG, ZIHENG
ZIA, Aneeq

Assignees

Intuitive Surgical Operations, Inc.

Dates

Publication Date: 20260506
Application Date: 20240627

Claims (1)

Atty. Dkt: 135039-0109 (P06771-WO) CLAIMS What is claimed is: 1. A system comprising: one or more processors, coupled with memory to: receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system; receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system; determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream; and modify the data stream in response to determining that the non-surgical information is present in the data stream. 2. The system of claim 1, wherein determining whether the one or more tools are connected to the robotic medical system comprises identifying that there are not one or more tools that provide a data stream component of the data stream connected to the robotic medical system. 3. The system of claim 1, wherein determining whether the one or more tools are connected to the robotic medical system comprises identifying that at least one tool that either provides a data stream component of the data stream or that facilitates presenting of the data stream component on a display is connected to the robotic medical system. 4. The system of claim 1, wherein the one or more tools comprises a medical tool and a visualization tool, wherein the data stream comprises a first data stream component received from the medical tool and a second data stream component received from an image capture device, and wherein in response to determining that the medical tool is connected to the robotic medical system and the visualization tool is also connected to the robotic medical system, the one or more processors to: identify a portion of the data stream comprising the second data stream component, the second data stream component comprising the non-surgical information; modify the portion of the data stream comprising the second data stream component; and Atty. Dkt: 135039-0109 (P06771-WO) store the data stream upon modification. 5. The system of claim 1, wherein the one or more tools comprises a medical tool, wherein in response to determining that the medical tool is not connected to the robotic medical system, the one or more processors to classify each frame of the data stream using a machine learning model as an in-body frame indicative of a region inside a patient’s body or an out-of-body frame indicative of a region outside of the patient’s body. 6. The system of claim 5, wherein the one or more processors to: modify each of the out-of-body frame in the data stream; and store the data stream upon modification. 7. The system of claim 5 or 6, wherein the one or more tools further comprises a visualization tool, and wherein the one or more processors to: determine that the visualization tool is connected to the robotic medical system; identify a portion of the data stream comprising a data stream component received from an image capture device; modify the portion of the data stream; and store the data stream upon modification. 8. The system of any of claims 1-7, wherein modification of the data stream comprises at least one of blurring, redacting, or deleting a portion of the data stream comprising the non- surgical information or blurring, redacting, or deleting textual information associated with the non-surgical information from the data stream. 9. The system of any of claims 1-7, wherein modification of the data stream comprises blurring, redacting, or deleting protected health information of the non-surgical information from the data stream. 10. The system of claim 1, wherein the one or more processors to generate a confidence score based on the event stream, the confidence score indicative of a likelihood of presence of the non-surgical information in the data stream. 11. A non-transitory computer-readable media comprising computer-readable instructions Atty. Dkt: 135039-0109 (P06771-WO) stored thereon that when executed by one or more processors cause the one or more processors to: receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system; receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system; determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream; and modify the data stream in response to determining that the non-surgical information is present in the data stream. 12. The non-transitory computer-readable media of claim 11, wherein determining whether the one or more tools are connected to the robotic medical system comprises identifying that there are not one or more tools that provide a data stream component of the data stream connected to the robotic medical system. 13. The non-transitory computer-readable media of claim 11, wherein determining whether the one or more tools are connected to the robotic medical system comprises identifying that at least one tool that either provides a data stream component of the data stream or that facilitates presenting of the data stream component on a display is connected to the robotic medical system 14. The non-transitory computer-readable media of claim 11, wherein the one or more tools comprises a medical tool and a visualization tool, wherein the data stream comprises a first data stream component received from the medical tool and a second data stream component received from an image capture device, and wherein in response to determining that the medical tool is connected to the robotic medical system and the visualization tool is also connected to the robotic medical system, the one or more processors to: identify a portion of the data stream comprising the second data stream component, the second data stream component comprising the non-surgical information; modify the portion of the data stream comprising the second data stream component; and store the data stream upon modification. Atty. Dkt: 135039-0109 (P06771-WO) 15. The non-transitory computer-readable media of claim 11, wherein the one or more tools comprises a medical tool, wherein in response to determining that the medical tool is not connected to the robotic medical system, the one or more processors to classify each frame of the data stream using a machine learning model as an in-body frame indicative of a region inside a patient’s body or an out-of-body frame indicative of a region outside of the patient’s body. 16. The non-transitory computer-readable media of claim 15, wherein the one or more processors further execute computer-readable instructions to: modify each of the out-of-body frame in the data stream; and store the data stream upon modification. 17. The non-transitory computer-readable media of claim 15 or 16, wherein the one or more processors further execute computer-readable instructions to: determine that the visualization tool is connected to the robotic medical system; identify a portion of the data stream comprising a data stream component received from an image capture device; modify the portion of the data stream; and store the data stream upon modification. 18. A system comprising: one or more processors, coupled with memory to: receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system; receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system; determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is absent from the data stream; and store the data stream without modification in response to determining that the non-surgical information is absent from the data stream. 19. The system of claim 18, wherein the one or more tools comprises a medical tool and a visualization tool, and wherein the one or more processors to: determine that the medical tool is connected to the robotic medical system; Atty. Dkt: 135039-0109 (P06771-WO) determine a visualization tool is not connected to the robotic medical system; and determine that the non-surgical information is absent from the data stream in response to determining that the visualization tool is not connected to the robotic medical system and the medical tool connected to the robotic medical system. 20. The system of claim 18, wherein the one or more tools comprises a medical tool and a visualization tool, and wherein the one or more processors to: determine that the medical tool is not connected to the robotic medical system; classify each frame of the data stream using a machine learning model as an in-body frame indicative of a region inside a patient’s body or an out-of-body frame indicative of a region outside of the patient’s body in response to determining that the medical tool is not connected to the robotic medical system; determine that each frame of the data stream is an in-body frame; determine that the visualization tool is not connected to the robotic medical system; and determine that the non-surgical information is absent from the data stream in response to determining that the visualization tool is not connected to the robotic medical system and each frame of the data stream is an in-body frame.

Description

Atty. Dkt: 135039-0109 (P06771-WO) SYSTEM AND METHOD FOR DETECTING AND REMOVING NON-SURGICAL DATA BACKGROUND [0001] This application claims the benefit of, and priority to, under 35 U.S.C. § 119, U.S. Provisional Patent Application No.63/511,596, filed June 30, 2023, which is hereby incorporated by reference herein in its entirety. BACKGROUND [0002] Surgical procedures may involve capturing imagery such as video feeds from a variety of viewpoints. For example, in some instances, at least part of the surgical procedure may be performed with a computer-assisted robotic medical system. A medical tool, such as an imaging device like an endoscope, may be used in the robotic medical system to provide imagery. Data sources such as cameras, sensors, etc. may also be located at various viewpoints in the surgical facility to capture and provide imagery of various aspects of the surgical procedure. The captured imagery from the surgical procedure may be processed in various ways. SUMMARY [0003] This disclosure is directed to a system and method for recognizing and scrubbing non-surgical information from a data stream, such as one or more surgical videos. In particular, this disclosure uses multiple data stream components (e.g., multiple video streams) and an event stream to accurately detect and scrub non-surgical information from the data stream. The event stream may determine when a medical tool is installed in a computer- assisted system and whether a visualization tool (e.g., an external visualization tool different from the medical tool or a user interface (“UI”) overlay) receiving data stream components from sources other than the medical tool is connected to the computer-assisted system. When the medical tool is installed and the visualization tool is not connected to the computer-assisted system, the data stream may be considered to include only relevant surgical information that may be useful to preserve. However, when the medical tool is installed in the computer- assisted system and the visualization tool is also connected to the computer-assisted system, the data stream may include both surgical and non-surgical information. The non-surgical information may be accurately identified and scrubbed from the data stream, preserving the relevant surgical information, and removing the irrelevant information that may potentially Atty. Dkt: 135039-0109 (P06771-WO) contain personal or sensitive data. When the medical tool is not installed in the computer- assisted system, a machine learning model may be used to analyze the data stream and classify each frame in the data stream as in-body or out-of-body. Based on the classification and whether the visualization tool is connected to the computer-assisted system or not, any identified non-surgical information may be removed from the data stream. [0004] At least one aspect is directed to a system having one or more processors to receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream, and modify the data stream in response to determining that the non- surgical information is present in the data stream. [0005] At least one aspect is directed to a non-transitory computer-readable media having computer-readable instructions stored thereon. The computer-readable instructions when executed by one or more processors cause the one or more processors to receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream, and modify the data stream in response to determining that the non- surgical information is present in the data stream. [0006] At least one aspect is directed to a system having one or more processors to receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is absent from the data stream, and store the data stream without modification in response to determining that the non-surgical information is absent excluded from the data stream. [0007] These and other aspects and implementations are discussed in detail below. The foregoing informati