Search

CN-122005083-A - Instrument pose control method and system based on four-pose optical acquisition

CN122005083ACN 122005083 ACN122005083 ACN 122005083ACN-122005083-A

Abstract

The application discloses an instrument pose control method and system based on four-position pose optical acquisition, wherein the method comprises the steps of synchronously acquiring multi-source initial data, reconstructing a dynamic three-dimensional digital twin body in real time in a virtual space, outputting an operation intention prediction result through a space-time network, simultaneously resolving to obtain a predicted pose sequence of an instrument in a future preset time window, accurately sensing a real-time relative position relationship between the instrument and a key anatomical structure, constructing a dynamically updated safety boundary field, starting an adaptive impedance controller and dynamically adjusting impedance parameters of the instrument when the future moment of the instrument is predicted to invade the safety boundary field or when the real-time force feedback data is abnormal, and generating an auxiliary guiding force instruction or a resistance instruction. The present disclosure fundamentally improves the safety, accuracy and accessibility of complex surgical procedures through multi-modal sensing, digital twin reconstruction, spatiotemporal prediction, dynamic boundary construction, adaptive control, and on-line learning depth coupling.

Inventors

  • ZHANG XIAOBO
  • LI WEI
  • LI JUN

Assignees

  • 安徽米度智能科技有限公司

Dates

Publication Date
20260512
Application Date
20260227

Claims (10)

  1. 1. The utility model provides an instrument pose control method based on four-way pose optical acquisition, which is characterized by comprising the following steps: Step S10, synchronously acquiring multi-source initial data, carrying out fusion processing on the multi-source initial data and generating a multi-mode fusion data packet, wherein the multi-source initial data comprises two-dimensional RGB images and three-dimensional point cloud data of an operation area and an instrument which are synchronously acquired by four optical acquisition devices along the orthogonal direction, nine-axis inertial data of the instrument which are acquired by a miniature inertial measurement unit, and force feedback data in the interaction process of the instrument and tissues, which are acquired by a six-dimensional force/moment sensor carried by the tail end of the instrument; S20, reconstructing a dynamic three-dimensional digital twin body in real time in a virtual space based on the multi-mode fusion data packet, wherein the dynamic three-dimensional digital twin body comprises a surgical instrument, a patient anatomy structure and an interaction relation between the surgical instrument and the patient anatomy structure; Step S30, inputting the state sequence of the dynamic three-dimensional digital twin body, the historical motion trail of the instrument and the real-time force feedback data into a space-time network after the pre-training is completed, outputting an operation intention prediction result through the space-time network, simultaneously resolving to obtain a predicted pose sequence of the instrument in a future preset time window, and accurately sensing the real-time relative position relationship between the instrument and a key anatomical structure; Step S40, constructing a dynamically updated safety boundary field based on the predicted pose sequence obtained by the calculation in the step S30 and the perceived real-time relative position relationship, wherein the safety boundary field is used for defining the safety range of the motion of the instrument; And step S50, when the future moment of the instrument is predicted to invade the safety boundary field or the real-time force feedback data is abnormal, starting the self-adaptive impedance controller and dynamically adjusting the impedance parameter of the self-adaptive impedance controller to generate an auxiliary guiding force instruction or a resistance instruction, wherein the auxiliary guiding force instruction or the resistance instruction is cooperated with the operation force of a doctor to drive the instrument to perform pose adjustment.
  2. 2. The four-pose optical acquisition-based instrument pose control method according to claim 1, wherein in step S10, the synchronous acquisition of the four optical acquisition devices is realized by a hardware triggered synchronous mode, and the two-dimensional RGB image and the three-dimensional point cloud data are subjected to preprocessing including denoising, distortion correction and coordinate system unification.
  3. 3. The method according to claim 1, wherein in step S20, the mechanical model of the virtual soft tissue is constructed based on a finite element method, and the initial value of the parameter of the mechanical model is adaptively generated based on pre-operative medical image data of the patient, wherein the medical image data includes a CT image or an MRI image.
  4. 4. The method for controlling the pose of an instrument based on four-dimensional optical acquisition according to claim 1, further comprising, after said step S50, a step S60 of: continuously comparing the deviation of the predicted interaction force of the digital twin body and the actual force feedback sensor data in the operation process; When the deviation exceeds a preset threshold, triggering a lightweight online learning cycle, and utilizing current operation data to finely adjust partial layers of the space-time network or tissue mechanics parameters of the physical engine so as to dynamically adapt the digital twin body to the individuation tissue characteristics of the current patient.
  5. 5. The method of claim 1 or 4, wherein the space-time network employs an encoder-decoder structure for processing a state sequence of a digital twin, and the decoder is configured to take a real-time operation signal of the instrument as a query vector and output a prediction sequence based on an anatomical structure and an instrument component most relevant to a current operation.
  6. 6. The four-dimensional optical acquisition-based instrument pose control method according to claim 1, wherein in the step S50, the impedance parameter adjustment of the adaptive impedance controller comprises: When the approach to the key anatomical structure is predicted, the damping of the guiding direction is gradually increased, and a virtual force field channel is rendered in a display module of the virtual space through augmented reality; When high frequency tremors or sudden increases in force feedback data are detected, the stiffness of the steering direction is instantaneously reduced and a high frequency filter is enabled for actively suppressing unstable operation.
  7. 7. The method according to claim 4, wherein in the step S60, a model parameter initial value capable of quickly adapting to a new task is obtained based on a meta-learning optimizer pre-trained on a large amount of off-line surgical data, so that the fine tuning process can quickly converge under limited surgical real-time data.
  8. 8. An instrument pose control system based on four-way pose optical acquisition for implementing the method according to any of claims 1 to 7, comprising: The multi-mode sensing module comprises four high-frame-rate optical cameras, an IMU sensor arranged in the instrument and a six-dimensional force/moment sensor at the tail end of the instrument; The digital twin calculation engine module is used for running a physical engine and a real-time three-dimensional reconstruction algorithm; The intelligent perception and prediction module is internally provided with a pre-trained space-time network; A cooperative control module for implementing a prospective adaptive impedance control algorithm, and The man-machine cooperative interaction module comprises an instrument with a force feedback function and an AR head-mounted display device, and is used for receiving doctor instructions and visualizing the digital twin body and the safety boundary.
  9. 9. The four-way pose optical acquisition based instrument pose control system according to claim 8, wherein the system supports multi-instrument co-operation, the digital twin calculation engine module independently constructs a sub-digital twin for each instrument and manages the motion rules and anti-collision logic between the instruments through a central coordinator.
  10. 10. The four-dimensional pose optical acquisition-based instrument pose control system according to claim 8, wherein the instrument of the human-computer interaction module can simulate the haptic sensation of a virtual boundary, the force sensation change of tissue incision and the haptic feedback of the interaction of the instrument with a virtual object such as a suture line according to the instruction of the self-adaptive impedance controller.

Description

Instrument pose control method and system based on four-pose optical acquisition Technical Field The invention relates to the technical field of medical instruments, in particular to an instrument pose control method and system based on four-pose optical acquisition. Background In the field of surgery, especially minimally invasive surgery, precise control of the pose of the instrument is directly related to the safety and effectiveness of the surgery. Traditional minimally invasive surgery relies on the experience of the surgeon and the precision of the hand manipulation, viewing the surgical field through an endoscope and manipulating the instruments to complete the surgical procedure. However, the two-dimensional image provided by the endoscope lacks depth information, so that a doctor is difficult to accurately judge the relative position relation between the instrument and the anatomical structure of a patient, the risk that the instrument collides with a key anatomical structure (such as a blood vessel and a nerve) is easy to occur, and on the other hand, the long-time operation can lead to hand fatigue of the doctor, operation tremble is generated, operation precision is affected, and the occurrence probability of operation complications is increased. Disclosure of Invention The embodiment of the invention aims to provide a four-position pose optical acquisition-based instrument pose control method and system, which can solve the problems in the prior art. In order to achieve the above purpose, the application adopts the following technical scheme: in one aspect, there is provided an instrument pose control method based on four-way pose optical acquisition, comprising: Step S10, synchronously acquiring multi-source initial data, carrying out fusion processing on the multi-source initial data and generating a multi-mode fusion data packet, wherein the multi-source initial data comprises two-dimensional RGB images and three-dimensional point cloud data of an operation area and an instrument which are synchronously acquired by four optical acquisition devices along the orthogonal direction, nine-axis inertial data of the instrument which are acquired by a miniature inertial measurement unit, and force feedback data in the interaction process of the instrument and tissues, which are acquired by a six-dimensional force/moment sensor carried by the tail end of the instrument; S20, reconstructing a dynamic three-dimensional digital twin body in real time in a virtual space based on the multi-mode fusion data packet, wherein the dynamic three-dimensional digital twin body comprises a surgical instrument, a patient anatomy structure and an interaction relation between the surgical instrument and the patient anatomy structure; Step S30, inputting the state sequence of the dynamic three-dimensional digital twin body, the historical motion trail of the instrument and the real-time force feedback data into a space-time network after the pre-training is completed, outputting an operation intention prediction result through the space-time network, simultaneously resolving to obtain a predicted pose sequence of the instrument in a future preset time window, and accurately sensing the real-time relative position relationship between the instrument and a key anatomical structure; Step S40, constructing a dynamically updated safety boundary field based on the predicted pose sequence obtained by the calculation in the step S30 and the perceived real-time relative position relationship, wherein the safety boundary field is used for defining the safety range of the motion of the instrument; And step S50, when the future moment of the instrument is predicted to invade the safety boundary field or the real-time force feedback data is abnormal, starting the self-adaptive impedance controller and dynamically adjusting the impedance parameter of the self-adaptive impedance controller to generate an auxiliary guiding force instruction or a resistance instruction, wherein the auxiliary guiding force instruction or the resistance instruction is cooperated with the operation force of a doctor to drive the instrument to perform pose adjustment. Preferably, in step S10, the synchronous acquisition of the four optical acquisition devices is implemented by a hardware triggered synchronous mode, and preprocessing including denoising, distortion correction and coordinate system unification is performed on the two-dimensional RGB image and the three-dimensional point cloud data. Preferably, in step S20, a mechanical model of the virtual soft tissue is constructed based on a finite element method, and an initial value of a parameter of the mechanical model is adaptively generated based on pre-operative medical image data of the patient, where the medical image data includes a CT image or an MRI image. Preferably, after the step S50, the method further includes a step S60: continuously comparing the deviation of the predicted interaction f