US-12616540-B2 - Imaging device control via multiple input modalities

US12616540B2US 12616540 B2US12616540 B2US 12616540B2US-12616540-B2

Abstract

Techniques are disclosed for imaging device control in a computer-assisted device that includes a repositionable structure configured to support an imaging device, and a control system configured to determine a position of a first reference point associated with a first input modality; in a first mode, determine a position of a target reference point for the imaging device based on the first reference point; in a second mode, determine a position of a second reference point associated with a second input modality, and determine the position of the target reference point based on the position of the first reference point and the position of the second reference point; determine a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and cause actuation of the repositionable structure based on the movement.

Inventors

Mohammad Sina Parastegari
Paul G. GRIFFITHS
Brandon D. Itkowitz
GORAN A. LYNCH

Assignees

Intuitive Surgical Operations, Inc.

Dates

Publication Date: 20260505
Application Date: 20211208

Claims (20)

1 . A computer-assisted device comprising: a first repositionable structure configured to support an imaging device; and a control system coupled to the first repositionable structure; wherein the control system is configured to: determine a position of a first reference point associated with a first input modality; in a first mode, determine a position of a target reference point for the imaging device based on at least the first reference point; in a second mode: determine a position of a second reference point associated with a second input modality, and determine the position of the target reference point based on at least the position of the first reference point and the position of the second reference point; determine a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and cause actuation of the first repositionable structure based on the determined movement; wherein: the first input modality comprises a hand-operated controller and the second input modality comprises one or more hand input sensors of a display unit, or the first input modality comprises the one or more head input sensors of the display unit and the second input modality comprises the hand-operated controller.
2 . The computer-assisted device of claim 1 , wherein: the first input modality comprises the hand-operated controller, and the second input modality comprises the one or more head input sensors of the display unit.
3 . The computer-assisted device of claim 1 , wherein: the first input modality comprises the one or more head input sensors of the display unit, the first reference point is offset from the display unit by being in front of the display unit in a direction away from an operator, the second input modality comprises the hand-operated controller, the hand-operated controller comprises a first input control and a second input control, the second reference point is at a midpoint between the first and second input controls, and the third reference point is offset from the imaging device by being in front of the imaging device in a direction of a field of view of the imaging device.
4 . The computer-assisted device of claim 1 , wherein: the imaging device and the first repositionable structure configured to support the imaging device together have a first plurality of degrees of freedom, the display unit and a second repositionable structure configured to support the display unit together have a second plurality of degrees of freedom different from the first plurality of degrees of freedom, and an input control of the hand-operated controller and a third repositionable structure configured to support the input control together have a third plurality of degrees of freedom different from the first plurality of degrees of freedom.
5 . The computer-assisted device of claim 4 , wherein the first plurality of degrees of freedom differ in number from the second plurality of degrees of freedom and the third plurality of degrees of freedom, or wherein the first plurality of degrees of freedom includes one or more degrees of freedom that is not matched by the second plurality of degrees of freedom or the third plurality of degrees of freedom.
6 . The computer-assisted device of claim 1 , further comprising: another repositionable structure configured to support a tool, wherein the control system is further configured to, in a third mode, cause actuation of the another repositionable structure to move the tool in response to input received via the second input modality.
7 . The computer-assisted device of claim 1 , wherein the control system is further configured to, in a third mode: determine the position of the second reference point associated with the second input modality, determine a position of a fourth reference point associated with a third input modality, and determine the position of the target reference point based on at least the position of the first reference point, the position of the second reference point, and the position of the fourth reference point.
8 . The computer-assisted device of claim 1 , wherein, to determine the position of the target reference point based on at least the position of the first reference point and the position of the second reference point, the control system is configured to: add a first scaled difference to a second scaled difference, the first scaled difference being a difference between the position of the first reference point and a baseline reference point of the first input modality scaled by a first scaling factor, and the second scaled difference being a difference between the position of the second reference point and a baseline reference point of the second input modality scaled by a second scaling factor.
9 . The computer-assisted device of claim 8 , wherein the control system is further configured to: determine the baseline reference point of the first input modality based on the position of the first reference point at an entry to the first mode; determine the baseline reference point of the second input modality based on the position of the second reference point at an entry to a third mode; and determine the baseline reference point of the imaging device based on a configuration of the imaging device at the entry to the first mode or the entry to the third mode.
10 . The computer-assisted device of claim 9 , wherein the control system is further configured to: in the third mode, determine the position of the second reference point associated with the second input modality, and determine the position of the target reference point based on at least the position of the second reference point.
11 . The computer-assisted device of claim 1 , wherein the control system is further configured to: determine whether the target reference point can be achieved using a virtual follower device; and in response to determining that the target reference point cannot be achieved: determine haptic feedback based on a difference between an achievable position of the third reference point and the target reference point, determine joint forces or torques for joints in a second repositionable structure configured to support the first input modality or a third repositionable structure configured to support the second input modality based on the haptic feedback, and cause actuation of the joints of the second repositionable structure or the third repositionable structure based on the determined joint forces or torques.
12 . The computer-assisted device of claim 1 , wherein the control system is further configured to, in response to determining that the first repositionable structure cannot achieve the determined movement: determine haptic feedback based on a difference between an achievable movement of the first repositionable structure and the determined movement; determine joint forces or torques for joints in a second repositionable structure configured to support the first input modality based on the haptic feedback; and cause actuation of the joints of the second repositionable structure based on the determined joint forces or torques.
13 . The computer-assisted device of claim 1 , wherein the control system is further configured to: in response to identifying a manual adjustment to the first input modality: set a first baseline reference point of the first input modality to the position of the first reference point at a completion of the manual adjustment to the first input modality, set a first correction distance based on a difference between the second reference point and a first baseline reference point of the second input modality, and based on a difference between the third reference point and a first baseline reference point of the imaging device, and modify the first correction distance by a ratcheting factor; or in response to identifying a manual adjustment to the second input modality: set a second baseline reference point of the second input modality to the position of the second reference point at a completion of the manual adjustment to the second input modality, set a second correction distance based on a difference between the first reference point and a second baseline reference point of the first input modality, and based on a difference between the third reference point and a second baseline reference point of the imaging device, and modify the second correction distance by the ratcheting factor; or in response to identifying a manual adjustment to the imaging device: set a third baseline reference point of the imaging device to the position of the third reference point at a completion of the manual adjustment to the imaging device, set a third correction distance based on a difference between the first reference point and a third baseline reference point of the first input modality, and based on a difference between the second reference point and a third baseline reference point of the second input modality, and modify the third correction distance by the ratcheting factor.
14 . A method of operating a computer-assisted device comprising a first repositionable structure and one or more processors, the one or more processors communicatively coupled to the first repositionable structure, the method comprising: determining, by the one or more processors, a position of a first reference point associated with a first input modality; in a first mode, determining, by the one or more processors, a position of a target reference point for an imaging device supported by a first repositionable structure of a computer-assisted device based on at least the first reference point; in a second mode: determining, by the one or more processors, a position of a second reference point associated with a second input modality, and determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point; determining, by the one or more processors, a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and causing, by the one or more processors, actuation of the first repositionable structure based on the determined movement, wherein: the first input modality comprises a hand-operated controller and the second input modality comprises one or more head input sensors of a display unit, or the first input modality comprises the one or more head input sensors of the display unit and the second input modality comprises the hand-operated controller.
15 . The method of claim 14 , wherein: the first input modality comprises the one or more head input sensors of the display unit, the first reference point is offset from the display unit by being in front of the display unit in a direction away from an operator, the second input modality comprises the hand-operated controller, the hand-operated controller comprises a first input control and a second input control, the second reference point is at a midpoint between the first and second input controls, and the third reference point is offset from the imaging device by being in front of the imaging device in a direction of a field of view of the imaging device.
16 . The method of claim 14 , further comprising: in a third mode, causing actuation of another repositionable structure to move a tool in response to input received via the second input modality, wherein the another repositionable structure is configured to support the tool.
17 . The method of claim 14 , wherein determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point comprises: adding a first scaled difference to a second scaled difference, the first scaled difference being a difference between the position of the first reference point and a baseline reference point of the first input modality scaled by a first scaling factor, and the second scaled difference being a difference between the position of the second reference point and a baseline reference point of the second input modality scaled by a second scaling factor.
18 . The method of claim 14 , further comprising, in response to identifying a manual adjustment to the first input modality: setting a first baseline reference point of the first input modality to the position of the first reference point at a completion of the manual adjustment to the first input modality, setting a first correction distance based on a difference between the second reference point and a first baseline reference point of the second input modality, and based on a difference between the third reference point and a first baseline reference point of the imaging device, and modifying the first correction distance by a ratcheting factor; or in response to identifying a manual adjustment to the second input modality: setting a second baseline reference point of the second input modality to the position of the second reference point at a completion of the manual adjustment to the second input modality, setting a second correction distance based on a difference between the first reference point and a second baseline reference point of the first input modality, and based on a difference between the third reference point and a second baseline reference point of the imaging device, and modifying the second correction distance by the ratcheting factor; or in response to identifying a manual adjustment to the imaging device: setting a third baseline reference point of the imaging device to the position of the third reference point at a completion of the manual adjustment to the imaging device, setting a third correction distance based on a difference between the first reference point and a third baseline reference point of the first input modality, and based on a difference between the second reference point and a third baseline reference point of the second input modality, and modifying the third correction distance by the ratcheting factor.
19 . One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which, when executed by one or more processors of a computer-assisted device comprising a first repositionable structure, are adapted to cause the one or more processors to perform a method comprising: determining a position of a first reference point associated with a first input modality, wherein the first input modality comprises one of a hand-operated controller or one or more head input sensors of a display unit; in a first mode, determining a position of a target reference point for an imaging device supported by a first repositionable structure of a computer-assisted device based on at least the first reference point; in a second mode: determining a position of a second reference point associated with a second input modality, and determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point; determining a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and causing actuation of the first repositionable structure based on the determined movement; wherein: the first input modality comprises a hand-operated controller and the second input modality comprises one or more head input sensors of a display unit, or the first input modality comprises the one or more head input sensors of the display unit and the second input modality comprises the hand-operated controller.
20 . The one or more non-transitory machine-readable media of claim 19 , wherein: the first input modality comprises the one or more head input sensors of the display unit, the first reference point is offset from the display unit by being in front of the display unit in a direction away from an operator, the second input modality comprises the hand-operated controller, the hand-operated controller comprises a first input control and a second input control, the second reference point is at a midpoint between the first and second input controls, and the third reference point is offset from the imaging device by being in front of the imaging device in a direction of a field of view of the imaging device.

Description

RELATED APPLICATIONS This application is a U.S. National Stage patent application of International Patent Application No. PCT/US2021/062466, filed Dec. 8, 2021, and claims the benefit to U.S. Provisional Application No. 63/123,939, filed Dec. 10, 2020, and entitled “Imaging Device Control via Multiple Input Modalities,” each of these related applications is incorporated by reference herein. TECHNICAL FIELD The present disclosure relates generally to electronic devices, and more particularly to control of devices with repositionable imaging devices. BACKGROUND More and more devices are being replaced with computer-assisted electronic devices. This is especially true in industrial, entertainment, educational, and other settings. As a medical example, the hospitals of today have large arrays of electronic devices being found in operating rooms, interventional suites, intensive care wards, emergency rooms, and/or the like. Many of these electronic devices may be capable of autonomous or semi-autonomous motion. It is also common for personnel to control the motion and/or operation of electronic devices using one or more input devices located at a user control system. As a specific example, minimally invasive, robotic telesurgical systems permit surgeons to operate on patients from bedside or remote locations. Telesurgery refers generally to surgery performed using surgical systems where the surgeon uses some form of remote control, such as a servomechanism, to manipulate surgical instrument movements rather than directly holding and moving the instruments by hand. When an electronic device is used to perform a task at a worksite, one or more imaging devices (e.g., an endoscope, an optical camera, and/or an ultrasound probe) can capture images of the worksite that provide visual feedback to an operator who is monitoring and/or performing the task. The imaging device(s) may also be controllable to update a view of the worksite that is provided, via a display unit, to the operator. For example, the imaging device(s) could be attached to a repositionable structure that includes two or more links coupled together by one or more joints, where the repositionable structure can be moved (including through internal reconfiguration) to update a position and/or orientation of the imaging device at the worksite. In such a case, movement of the imaging device(s) may be controlled by the operator, another person, or automatically, and enable the view of the worksite to be changed. One approach for controlling an imaging device is to move the imaging device to follow the motion of a display unit. For example, the head motion of an operator can be tracked via a sensor system and used to control the motion of the imaging device. Another approach for controlling an imaging device is to move the imaging device to follow the motion of a hand-operated controller. For example, the operator could control a hand-operated controller with one input device operated by each hand of the operator. These approaches are used separately, and each has its own advantages and disadvantages. Accordingly, improved methods and systems for controlling repositionable imaging devices are desirable. SUMMARY Consistent with some embodiments, a computer-assisted device includes a first repositionable structure configured to support an imaging device; and a control system coupled to the first repositionable structure. The control system is configured to: determine a position of a first reference point associated with a first input modality, in a first mode, determine a position of a target reference point for the imaging device based on at least the first reference point, in a second mode, determine a position of a second reference point associated with a second input modality, and determine the position of the target reference point based on at least the position of the first reference point and the position of the second reference point, determine a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point, and cause actuation of the first repositionable structure based on the determined movement. Consistent with some embodiments, a computer-assisted device includes a first repositionable structure configured to support an imaging device, and a control system coupled to the first repositionable structure. The control system is configured to: determine a position of a first reference point associated with a first input modality, determine a position of a first target reference point based on at least the position of the first reference point, determine whether the first repositionable structure can be actuated to move the imaging device such that a third reference point associated with the imaging device moves to the first target reference point, determine a position of a second reference point associated with a second input modali