Search

EP-4342409-B1 - FORCE SENSE DISPLAY DEVICE AND PROGRAM

EP4342409B1EP 4342409 B1EP4342409 B1EP 4342409B1EP-4342409-B1

Inventors

  • NISHIHARA, Teruyuki

Dates

Publication Date
20260506
Application Date
20210518

Claims (6)

  1. A force sense display device (50, 50A), comprising: an image obtainer (51) configured to obtain captured-image information that is output from an image-capturing section (35) configured to capture an image of an affected part; a force sense obtainer (52) configured to obtain force sense information comprising information on a magnitude of and a direction of a force acting on at least one surgical instrument (21R, 21L, 121R, 121L) for treatment of the affected part; a generator (53) configured to generate a force sense representation based on the force sense information, the force sense representation being a mode of representation to be displayed on a display section (12) and comprising a force amount representation (41R, 41L, 46R, 46L) that represents the magnitude of the force and a direction representation (42R, 42L) that represents the direction of the force; a detector (54, 54A) configured to detect a surgical instrument area (27R, 27L, 28R, 28L) where the surgical instrument (21R, 21L, 121R, 121L) is displayed in an image displayed on the display section (12), based on the captured-image information; and a superimposing section (55) configured to generate display information for display, on the display section (12), of the surgical instrument area (27R, 27L, 28R, 28L) with the force sense representation superimposed thereon, wherein the force sense information comprises information on a magnitude of and a direction of a first force acting on a shaft (22R, 22L) of the surgical instrument (21R, 21L, 121R, 121L) extending in a rod shape, wherein the generator (53) generates a first force sense representation comprising a first force amount representation (41R, 41L) that represents the magnitude of the first force and a first direction representation (42R, 42L) that represents the direction of the first force, wherein the detector (54, 54A) detects, as a first surgical instrument area (27R, 27L), an area where the shaft (22R, 22L) is displayed in the image displayed on the display section (12), and wherein the superimposing section (55) generates display information for display, on the display section (12), of the first surgical instrument area (27R, 27L) with the first force sense representation superimposed thereon, wherein the force sense information comprises information on a magnitude of and a direction of a second force applied to an end effector (23R, 23L) that is provided on an end of the shaft (22R, 22L) and used for treatment of the affected part, wherein the generator (53) generates a second force sense representation comprising a second force amount representation (46R, 46L) that represents the magnitude of the second force and a second direction representation that represents the direction of the second force, wherein the detector (54, 54A) detects, as a second surgical instrument area (28R, 28L), an area where the end effector (23R, 23L) is displayed in the image displayed on the display section (12), and wherein the superimposing section (55) generates display information for display, on the display section (12), of the second surgical instrument area (28R, 28L) with the second force sense representation superimposed thereon.
  2. The force sense display device (50, 50A) according to claim 1, wherein the generator (53) generates the force sense representation comprising the force amount representation (41R, 41L, 46R, 46L) that represents the magnitude of the force using at least one of a color type, a color brightness/darkness, or a transparency.
  3. The force sense display device (50, 50A) according to claim 1 or 2, wherein the generator (53) generates the force sense representation comprising the force amount representation (41R, 41L, 46R, 46L) that represents the magnitude of the force using a numerical value.
  4. The force sense display device (50, 50A) according to claim 3, wherein the generator (53) generates the force sense representation comprising the force amount representation (41R, 41L, 46R 46L) that represents the magnitude of the force using a size of the numerical value to be displayed.
  5. The force sense display device (50A) according to any one of claims 1 to 4, further comprising a storage section (57A) configured to store at least one of a color tone or a pattern of the surgical instrument (121R, 121L), wherein the detector (54A) detects the surgical instrument area (27R, 27L, 28R, 28L) in the image displayed on the display section (12), based on the at least one of the color tone or the pattern of the surgical instrument (121R, 121L) stored in the storage section (57A).
  6. A program for causing a computer to execute: an image obtaining function of obtaining captured-image information that is output from an image-capturing section (35) configured to capture an image of an affected part; a force sense obtaining function of obtaining force sense information comprising information on a magnitude of and a direction of a force acting on at least one surgical instrument (21R, 21L, 121R, 121L) for treatment of the affected part; a generating function of generating a force sense representation based on the force sense information, the force sense representation being a mode of representation to be displayed on a display section (12) and comprising a force amount representation (41R, 41L, 46R, 46L) that represents the magnitude of the force and a direction representation (42R, 42L) that represents the direction of the force; a detecting function of detecting a surgical instrument area (27R, 27L, 28R, 28L) where the surgical instrument (21R, 21L, 121R, 121L) is displayed in an image displayed on the display section, based on the captured-image information; and a superimposing function of generating display information for display, on the display section (12), of the surgical instrument area (27R, 27L, 28R, 28L) with the force sense representation superimposed thereon, wherein the force sense information comprises information on a magnitude of and a direction of a first force acting on a shaft (22R, 22L) of the surgical instrument (21R, 21L, 121R, 121L) extending in a rod shape, wherein the generating function generates a first force sense representation comprising a first force amount representation (41R, 41L) that represents the magnitude of the first force and a first direction representation (42R, 42L) that represents the direction of the first force, wherein the detecting function detects, as a first surgical instrument area (27R, 27L), an area where the shaft (22R, 22L) is displayed in the image displayed on the display section (12), and wherein the superimposing function generates display information for display, on the display section (12), of the first surgical instrument area (27R, 27L) with the first force sense representation superimposed thereon, wherein the force sense information comprises information on a magnitude of and a direction of a second force applied to an end effector (23R, 23L) that is provided on an end of the shaft (22R, 22L) and used for treatment of the affected part, wherein the generating function generates a second force sense representation comprising a second force amount representation (46R, 46L) that represents the magnitude of the second force and a second direction representation that represents the direction of the second force, wherein the detecting function detects, as a second surgical instrument area (28R, 28L), an area where the end effector (23R, 23L) is displayed in the image displayed on the display section (12), and wherein the superimposing function generates display information for display, on the display section (12), of the second surgical instrument area (28R, 28L) with the second force sense representation superimposed thereon.

Description

TECHNICAL FIELD The present disclosure relates to a force sense display device, an unclaimed force sense display method, and a program. BACKGROUND ART In endoscopic surgery, laparoscopic surgery, and the like, use of surgical tools such as an endoscope, a laparoscope, and forceps, is becoming widespread. Hereinafter, a surgical tool such as an endoscope, a laparoscope, or forceps is also referred to as a "surgical instrument". Endoscopic surgery, laparoscopic surgery, and the like, using the surgical instrument, are also referred to as "endoscopic surgery and the like". In endoscopic surgery and the like, in addition to surgeries using a surgical instrument held in the hand of an operator such as a doctor, there are also surgeries performed using a medical robot performing surgical assistance as disclosed in, for example, Patent Document 1. The surgery using a medical robot that performs surgical assistance is also referred to as a robotic-assisted surgery. PRIOR ART DOCUMENTS PATENT DOCUMENTS Patent Document 1: Japanese Unexamined Patent Application Publication No. 2001-104333Patent Document 2: US 2020/188045 A1Patent Documents 3: WO 2020/189110 A1Patent Document 4: WO 2020/176128 A1Patent Document 5: WO 2020/189120 A1 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION In endoscopic surgery and the like, it is known that operators find it difficult to know a force exerted by a surgical instrument on tissues such as organs. For example, it is known that operators find it difficult to know forces such as a force with which a surgical instrument presses tissue, and a force with which a surgical instrument holds tissue. As a result, more force than intended is applied to the tissue, which may lead to damage to the tissue. Hereinafter, information relating to a force exerted by a surgical instrument on tissues such as organs is referred to as "force amount information". As a method for an operator to know the force amount information described above, it is known to use force amount display equipment specialized for displaying the force amount information to the operator, separately from image display equipment for displaying images captured by an endoscope. This use method requires the operator to move a line of sight from the image display equipment to the force amount display equipment, in order for the operator to know the force amount information. Thus, there is a concern that tissue may be accidentally damaged during movement of the line of sight. Further, a method is known of displaying, on an image display unit, force amount information superimposed. In this method, the force amount information is superimposed on a portion where tissue is displayed on the image display unit, thereby to hide the tissue, thus making it difficult to visually recognize the hidden tissue. Thus, there is a concern that the tissue, which is difficult to be visually recognized, may be damaged accidentally. Patent Document 1 described above discloses a technique of displaying force amount information superimposed on a part of an instrument corresponding to the surgical instrument displayed on an image display unit. Specifically, Patent Document 1 discloses a technique of superimposing and displaying an indicator representing a magnitude of a force to hold tissue, and information on a joint angle of the instrument and on an angle of rotation thereof. There is a concern that the information to be displayed in this case is insufficient and thus it is difficult to predict damage to the tissue. Patent Document 2 describes a robotic surgical system comprising a visual display including a force feedback meter, and computer system which can position a surgical tool, an end effector relative to a surgical site, and an image capture device, the images thereof being displayed by the visual display. Patent Document 3 describes a medical robot system with a control device which can create image information about an external force as an image in association with the location where the external force was measured, and a display device which displays the image information. Patent Document 4 describes methods for measuring a force by a tracking system, wherein numerical values for the magnitude and/or direction may be displayed to a user, and if the user is wearing an augmented reality headset, feedback information pertaining to the magnitude may include a visual indication overlaid over the tool. Patent Document 5 describes a force sense display device having an input unit which is connected to a shaft sensor that measures an external force applied to a shaft, a joint sensor that measures the external force applied to the joint portion, a surgical tool sensor that measures the opening and closing force applied to the outside by a surgical tool and an image generating unit which generates image information based on video and image signals input from an endoscope unit and signals input from the shaft sensor, the joint sensor, and the