US-12616533-B2 - Methods and systems for using voice input to control a surgical robot

Abstract

Methods, apparatuses, and systems for using speech input to control a surgical robot are disclosed. A surgical robot is disclosed that can be controlled by a surgeon using speech input in a conversational manner. The surgical robot is provided either general commands or specific instructions, assessing whether the instructions can be completed within the capabilities of the available hardware and resources, and seeking approval from the surgeon prior to executing the instructions. Alternatively, the embodiments disclosed allow the surgeon to perform an action that cannot be safely completed by the surgical robot.

Inventors

• Jeffrey Roh
• Justin Esterberg
• John Cronin
• SETH CRONIN
• Michael John Baker

Assignees

• IX INNOVATION LLC

Dates

Publication Date

20260505

Application Date

20220831

Claims (17)

1. 1 . A computer-implemented method comprising: receiving speech input associated with a surgical procedure; receiving anatomical data: classifying one or more features in the anatomical data; determining at least one instruction from the speech input using a speech engine trained using data associated with the surgical procedure, wherein the at least one instruction is determined based on the classified one or more features; generating one or more actions to be performed by a surgical robot, the one or more actions based on the at least one instruction; determining, by the surgical robot, that the surgical robot is capable of performing the one or more actions of the surgical procedure; and responsive to determining that the surgical robot is capable of performing the one or more actions, performing, by the surgical robot, the one or more actions.
2. 2 . The method of claim 1 , comprising: determining that the one or more actions are incompatible with a surgical tool being used in the surgical procedure; and responsive to determining that the one or more actions are incompatible, transferring, by the surgical robot, control of the surgical procedure to a surgeon.
3. 3 . The method of claim 1 , comprising: identifying an authorized user associated with the speech input; and authorizing the one or more actions to be performed by the robot based on the identification.
4. 4 . The method of claim 1 , comprising: receiving an environment acoustic signal from a surgical microphone positioned to detect noise from the surgical procedure; receiving an acoustic signal from a user interface; and comparing the environment acoustic signal and the acoustic signal from the user interface to actively cancel noise in the acoustic signal from the user interface.
5. 5 . The method of claim 1 , wherein the anatomical data includes image data, the method further comprising using a machine learning model and/or a deep learning network to classify the one or more features in the image data.
6. 6 . The method of claim 1 , wherein the speech engine comprises a user-specific speech processing module trained using speech data from a user.
7. 7 . A surgical robot comprising: one or more computer processors; and a non-transitory computer-readable storage medium storing computer instructions, which when executed by the one or more computer processors, cause the surgical robot to: receive speech input associated with a surgical procedure; receive anatomical data: classify one or more features in the anatomical data: determine at least one instruction from the speech input using a speech engine trained using data associated with the surgical procedure; wherein the at least one instruction is determined based on the classified one or more features; generate one or more actions to be performed by a surgical robot, the one or more actions based on the at least one instruction; determine, by the surgical robot, that the surgical robot is capable of performing the one or more actions of the surgical procedure; and responsive to determining that the surgical robot is capable of performing the one or more actions, perform, by the surgical robot, the one or more actions.
8. 8 . The surgical robot of claim 7 , wherein the computer instructions cause the surgical robot to: determine that the one or more actions are incompatible with a surgical tool being used in the surgical procedure; and responsive to determining that the one or more actions are incompatible, transfer, by the surgical robot, control of the surgical procedure to a surgeon.
9. 9 . The surgical robot of claim 7 , wherein the computer instructions cause the surgical robot to: identify an authorized user associated with the speech input; and authorize the one or more actions to be performed by the robot based on the identification.
10. 10 . The surgical robot of claim 7 , wherein the computer instructions cause the surgical robot to: receive an environment acoustic signal from a surgical microphone positioned to detect noise from the surgical procedure; receive an acoustic signal from a user interface; and compare the environment acoustic signal and the acoustic signal from the user interface to actively cancel noise in the acoustic signal from the user interface.
11. 11 . The surgical robot of claim 7 , wherein the anatomical data includes image data, the computer instructions, which when executed by the one or more computer processors, cause the surgical robot to use a machine learning model and/or a deep learning network to classify the one or more features in the image data to identify anatomical features.
12. 12 . The surgical robot of claim 7 , wherein the speech engine comprises a user-specific speech processing module trained using speech data from a user.
13. 13 . A robotic surgical system comprising: a non-transitory computer-readable storage medium storing computer instructions, which when executed by one or more computer processors, cause the robotic surgical system to: receive speech input associated with a surgical procedure; receive anatomical data; classify one or more features in the anatomical data: determine at least one instruction from the speech input using a speech engine trained using data associated with the surgical procedures wherein the at least one instruction is determined based on the classified one or more features; generate one or more actions to be performed by a surgical robot, the one or more actions based on the at least one instruction; determine, by the surgical robot, that the surgical robot is capable of performing the one or more actions of the surgical procedure; and responsive to determining that the surgical robot is capable of performing the one or more actions, perform, by the surgical robot, the one or more actions.
14. 14 . The robotic surgical system of claim 13 , wherein the computer instructions cause the robotic surgical system to: determine that the one or more actions are incompatible with a surgical tool being used in the surgical procedure; and responsive to determining that the one or more actions are incompatible, transfer, by the surgical robot, control of the surgical procedure to a surgeon.
15. 15 . The robotic surgical system of claim 13 , wherein the computer instructions cause the robotic surgical system to: identify an authorized user associated with the speech input; and authorize the one or more actions to be performed by the surgical robot based on the identification.
16. 16 . The robotic surgical system of claim 13 , wherein the computer instructions cause the robotic surgical system to: receive an environment acoustic signal from a surgical microphone positioned to detect noise from the surgical procedure; receive an acoustic signal from a user interface; and compare the environment acoustic signal and the acoustic signal from the user interface to actively cancel noise in the acoustic signal from the user interface.
17. 17 . The robotic surgical system of claim 13 , wherein the anatomical data includes image data, the computer instructions, which when executed by the one or more computer processors, cause the surgical robot to use a machine leaming model and/or a deep learning network to classify the one or more features in the image data to identify anatomical features.

Description

CROSS-REFERENCE TO RELATED APPLICATION The application is a continuation of U.S. patent application Ser. No. 17/568,362, filed Jan. 4, 2022, which is incorporated herein by reference in its entirety. TECHNICAL FIELD The present disclosure is generally related to automated and robotic surgical procedures and specifically to systems and methods for using voice input to control a surgical robot. BACKGROUND More than 200 million surgeries are performed worldwide each year, and recent reports reveal that adverse event rates for surgical conditions remain unacceptably high, despite traditional patient safety initiatives. Adverse events resulting from surgical interventions can be related to errors occurring before or after the procedure as well as technical surgical errors during the operation. For example, adverse events can occur due to (i) breakdown in communication within and among the surgical team, care providers, patients, and their families; (ii) delay in diagnosis or failure to diagnose; and (iii) delay in treatment or failure to treat. The risk of complications during surgery can include anesthesia complications, hemorrhaging, high blood pressure, a rise or fall in body temperature, etc. Such adverse events can further occur due to medical errors, infections, underlying physical or health conditions of the patient, reactions to anesthetics or other drugs, etc. Conventional methods for preventing wrong-site, wrong-person, wrong-procedure errors, or retained foreign objects are typically based on communication between the patient, the surgeon(s), and other members of the health care team. However, conventional methods are typically insufficient to prevent surgical errors and adverse events during surgery. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram illustrating an example surgical system, in accordance with one or more embodiments. FIG. 2 is a block diagram illustrating an example machine learning (ML) system, in accordance with one or more embodiments. FIG. 3 is a block diagram illustrating an example computer system, in accordance with one or more embodiments. FIG. 4A is a block diagram illustrating an example robotic surgical system, in accordance with one or more embodiments. FIG. 4B illustrates an example console of the robotic surgical system of FIG. 4A, in accordance with one or more embodiments. FIG. 5 is a schematic block diagram illustrating subcomponents of the robotic surgical system of FIG. 4A, in accordance with one or more embodiments. FIG. 6 is a block diagram illustrating an example robotic surgical system for using speech input to control a surgical robot, in accordance with one or more embodiments. FIG. 7 is a table illustrating an example procedure database, in accordance with one or more embodiments. FIG. 8 is a flow diagram illustrating an example process for using speech input to control a surgical robot, in accordance with one or more embodiments. FIG. 9 is a flow diagram illustrating an example process for using speech input to control a surgical robot, in accordance with one or more embodiments. FIG. 10 is a flow diagram illustrating an example process for using speech input to control a surgical robot, in accordance with one or more embodiments. DETAILED DESCRIPTION Embodiments of the present disclosure will be described more thoroughly from now on with reference to the accompanying drawings. Like numerals represent like elements throughout the several figures, and in which example embodiments are shown. However, embodiments can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. The examples set forth herein are non-limiting examples and are merely examples, among other possible examples. Throughout this specification, plural instances (e.g., “602”) can implement components, operations, or structures (e.g., “602a”) described as a single instance. Further, plural instances (e.g., “602”) refer collectively to a set of components, operations, or structures (e.g., “602a”) described as a single instance. The description of a single component (e.g., “602a”) applies equally to a like-numbered component (e.g., “602b”) unless indicated otherwise. These and other aspects, features, and implementations can be expressed as methods, apparatus, systems, components, program products, means or steps for performing a function, and in other ways. These and other aspects, features, and implementations will become apparent from the following descriptions. A surgeon sometimes needs the simultaneous use of multiple tools, exceeding the capacity of their two hands. Traditionally, nurses and other doctors assist the surgeon, following their direction. Recently, surgical robots are being used to replace additional personnel in assisting a surgeon; however, unlike other humans, the surgical robots do not accept speech commands. Surgical robots are traditionally controlled by physical interfaces, such as a joystick or other pr