US-12626467-B2 - Remote medical guidance method and system based on virtual reality

US12626467B2US 12626467 B2US12626467 B2US 12626467B2US-12626467-B2

Abstract

Disclosed are a remote medical guidance method and system based on virtual reality, belonging to the technical field of medical guidance. The method includes: obtaining an action sequence of remote guidance personnel for medical operations; and generating a three-dimensional operation model based on the action sequence and displaying the three-dimensional operation model in an operation live scene. The present disclosure achieves remote guidance of medical operations based on virtual reality, eliminating the need to relocate patients or medical personnel and avoiding missing the optimal treatment time window.

Inventors

Zhaowen ZONG
Haoyang Yang
Chenglin DAI
Hao Li
Hai Nan
Wenjuan DU
Renqing JIANG
Yijun JIA
Can Chen
Xin Zhong
Junting Cai
Ye Wei

Assignees

ARMY MEDICAL UNIVERSITY, PEOPLE'S LIBERATION ARMY, PRC
CHONGQING UNIVERSITY OF TECHNOLOGY

Dates

Publication Date: 20260512
Application Date: 20240313
Priority Date: 20231220

Claims (8)

1 . A remote medical guidance method based on virtual reality, comprising the following steps: obtaining an action sequence of remote guidance personnel for medical operations; and generating a three-dimensional operation model based on the action sequence and displaying the three-dimensional operation model in an operation live scene; wherein said obtaining the action sequence of the remote guidance personnel for the medical operations specifically comprises; obtaining RGB images at each moment during a medical operation process by the remote guidance personnel: obtaining depth images at each moment during the medical operation process by the remote guidance personnel: processing the RGB images at each moment to obtain plane position coordinates of key points at different moments, wherein the key points are hand joints of the remote guidance personnel: processing the depth images at each moment to obtain depth position coordinates of the key points at different moments; and generating the action sequence based on the plane position coordinates and the depth position coordinates of the key points at different moments.
2 . The remote medical guidance method based on virtual reality according to claim 1 , wherein said processing the RGB images at each moment to obtain the plane position coordinates of the key points at different moments specifically comprises: extracting a region of interest from the RGB images, wherein the region of interest is an area in which hands of the remote guidance personnel are located; performing edge recognition on the region of interest to obtain a hand contour; based on the hand contour and a hand skeletal structure, identifying the hand joints as the key points; determining position information of the identified key points in the RGB images; and performing coordinate transformation on the position information of the key points in the RGB images to determine the plane position coordinates of the key points, wherein the plane position coordinates comprise x-axis coordinates and y-axis coordinates in a world coordinate system.
3 . The remote medical guidance method based on virtual reality according to claim 1 , wherein said processing the depth images at each moment to obtain the depth position coordinates of the key points at different moments specifically comprises: aligning the depth image with the RGB image at the same moment based on intrinsic parameters and extrinsic parameters of an RGB camera and a depth camera to obtain aligned depth images; based on position information of the key points in the RGB images, determining depth information of the key points at corresponding positions in the aligned depth images; and performing coordinate transformation on the depth information of the key points at the corresponding positions in the aligned depth images to determine the depth position coordinates of the key points, wherein the depth position coordinates are z-axis coordinates in a world coordinate system.
4 . The remote medical guidance method based on virtual reality according to claim 1 , wherein said generating the three-dimensional operation model based on the action sequence specifically comprises: locating, connecting, and rendering the key points at each moment based on the plane position coordinates and the depth position coordinates of the key points at each moment, to generate three- dimensional hand pose models for each moment, and arranging and connecting the three-dimensional hand pose models for each moment in chronological order to generate the three-dimensional operation model.
5 . A remote medical guidance system based on virtual reality, comprising a remote terminal and an on-site medical terminal, wherein the on-site medical terminal comprises an augmented reality (AR) device worn by medical personnel; the remote terminal and the on-site medical terminal are connected; the remote terminal is configured to obtain an action sequence of remote guidance personnel for medical operations; and the on-site medical terminal is configured to generate a three-dimensional operation model based on the action sequence and display the three-dimensional operation model in an operation live scene; wherein the remote terminal comprises an RGB camera, a depth camera, and a data processing module: the RGB camera is connected to the data processing module, and is configured to obtain RGB images at each moment during a medical operation process by the remote guidance personnel; the depth camera is connected to the data processing module, and is configured to obtain depth images at each moment during the medical operation process by the remote guidance personnel; and the data processing module is connected to the on-site medical terminal, and is configured to process the RGB images at each moment to obtain plane position coordinates of the key points at different moments; process the depth images at each moment to obtain depth position coordinates of the key points at different moments; and generate the action sequence based on the plane position coordinates and the depth position coordinates of the key points at different moments, wherein the key points are hand joints of the remote guidance personnel.
6 . The remote medical guidance system based on virtual reality according to claim 5 , wherein in terms of processing the RGB images at each moment to obtain the plane position coordinates of the key points at different moments, the data processing module is specifically configured to: extract a region of interest from the RGB images, wherein the region of interest is an area in which hands of the remote guidance personnel are located; perform edge recognition on the region of interest to obtain a hand contour; based on the hand contour and a hand skeletal structure, identify the hand joints as the key points; determine position information of the identified key points in the RGB images; and perform coordinate transformation on the position information of the key points in the RGB images to determine the plane position coordinates of the key points, wherein the plane position coordinates comprise x-axis coordinates and y-axis coordinates in a world coordinate system.
7 . The remote medical guidance system based on virtual reality according to claim 5 , wherein in terms of processing the depth images at each moment to obtain the depth position coordinates of the key points at different moments, the data processing module is specifically configured to: align the depth image with the RGB image at the same moment based on intrinsic parameters and extrinsic parameters of an RGB camera and a depth camera to obtain aligned depth images; based on position information of the key points in the RGB images, determine depth information of the key points at corresponding positions in the aligned depth images; and perform coordinate transformation on the depth information of the key points at the corresponding positions in the aligned depth images to determine the depth position coordinates of the key points, wherein the depth position coordinates are z-axis coordinates in a world coordinate system.
8 . The remote medical guidance system based on virtual reality according to claim 5 , wherein in terms of generating the three-dimensional operation model based on the action sequence, the on-site medical terminal is specifically configured to: locate, connect, and render the key points at each moment based on the plane position coordinates and the depth position coordinates of the key points at each moment, to generate three- dimensional hand pose models for each moment; and arrange and connect the three-dimensional hand pose models for each moment in chronological order to generate the three-dimensional operation model.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims the benefit and priority of Chinese Patent Application No. 2023117664418, filed with the China National Intellectual Property Administration on Dec. 20, 2023, the disclosure of which is incorporated by reference herein in its entirety as part of the present application. TECHNICAL FIELD The present disclosure relates to the technical field of medical guidance, and in particular, to a remote medical guidance method and system based on virtual reality. BACKGROUND In the field of healthcare, in the case of complex surgeries or emergency procedures that on-site medical personnel cannot handle, the usual approach is to either transfer the patient or bring personnel capable of performing the operation to the site. However, this method is time-consuming and labor-intensive, and may result in missing the optimal treatment time window. SUMMARY An objective of the present disclosure is to provide a remote medical guidance method and system based on virtual reality to achieve remote guidance without relocating patients or medical personnel. To achieve the above objective, the present disclosure provides the following technical solutions. The present disclosure provides a remote medical guidance method based on virtual reality, including the following steps: obtaining an action sequence of remote guidance personnel for medical operations; andgenerating a three-dimensional operation model based on the action sequence and displaying the three-dimensional operation model in an operation live scene. The present disclosure further provides a remote medical guidance system based on virtual reality. The system includes a remote terminal and an on-site medical terminal. The on-site medical terminal includes an augmented reality (AR) device worn by medical personnel. The remote terminal and the on-site medical terminal are connected. The remote terminal is configured to obtain an action sequence of remote guidance personnel for medical operations. The on-site medical terminal is configured to generate a three-dimensional operation model based on the action sequence and display the three-dimensional operation model in an operation live scene. According to specific embodiments provided in the present disclosure, the present disclosure has the following technical effects: Embodiments of the present disclosure provide a remote medical guidance method and system based on virtual reality. The method includes: obtaining an action sequence of remote guidance personnel for medical operations; and generating a three-dimensional operation model based on the action sequence and displaying the three-dimensional operation model in an operation live scene. The present disclosure achieves remote guidance of medical operations based on virtual reality, eliminating the need to relocate patients or medical personnel and avoiding missing the optimal treatment time window. BRIEF DESCRIPTION OF THE DRAWINGS To describe the technical solutions in embodiments of the present disclosure or in the prior art more clearly, the accompanying drawings required in the embodiments are briefly described below. Apparently, the accompanying drawings in the following description show merely some embodiments of the present disclosure, and other drawings can be derived from these accompanying drawings by those of ordinary skill in the art without creative efforts. FIG. 1 is a flowchart of a remote medical guidance method based on virtual reality according to an embodiment of the present disclosure; FIG. 2 is a schematic diagram of an image coordinate system according to an embodiment of the present disclosure; FIG. 3 is a schematic diagram of a camera coordinate system and a world coordinate system according to an embodiment of the present disclosure; FIG. 4 is a schematic diagram showing a relationship between the image coordinate system and the camera coordinate system according to an embodiment of the present disclosure; FIG. 5 is a schematic diagram showing a relationship between the camera coordinate system and the world coordinate system according to an embodiment of the present disclosure; and FIG. 6 is a schematic structural diagram of a remote medical guidance system based on virtual reality according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS The technical solutions in the embodiments of the present disclosure are described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are merely some rather than all of the embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure. An objective of the present disclosure is to provide a remote medical guidance method and system b