US-12622766-B2 - Visualization of medical environments with predetermined 3D models

Abstract

An object or person in a medical environment may be identified based on images of the medical environment. The identification may include determining an identifier associated with the object or the person, a position of the object or the person in the medical environment, and a three-dimensional (3D) shape/pose of the object or the person. Representation information that indicates at least the determined identifier, position in the medical environment, and 3D shape/pose of the object or the person may be generated and then used (e.g., by a visualization device) together with one or more predetermined 3D models to determine a 3D model for the object or the person identified in the medical environment and generate a visual depiction of at least the object or the person in the medical environment based on the determined 3D model and the position of the object or the person in the medical environment.

Inventors

• Abhishek Sharma
• Arun Innanje
• Terrence Chen

Assignees

• SHANGHAI UNITED IMAGING INTELLIGENCE CO., LTD.

Dates

Publication Date

20260512

Application Date

20230413

Claims (17)

1. 1 . A system, comprising: a computing device, wherein the computing device comprises one or more image sensors and one or more processors, the one or more image sensors configured to capture one or more images of a medical environment, the one or more processors configured to: identify, based on the one or more images, an object or a person in the medical environment, wherein, as a part of the identification, the computing device is configured to determine at least an identifier associated with the object or the person, a position of the object or the person in the medical environment, and a three-dimensional (3D) shape of the object or the person; and generate representation information that indicates at least the identifier associated with the object or the person, the position of the object or the person in the medical environment, and the 3D shape of the object or the person; and a visualization device, wherein the visualization device comprises at least one processor and at least one display device, the at least one processor configured to: obtain the representation information generated by the computing device; determine, based on the representation information obtained from the computing device and one or more predetermined 3D models, a 3D model for the object or the person identified in the medical environment; and generate a visual depiction of at least the object or the person in the medical environment based on the determined 3D model and the position of the object or the person indicated by the representation information; wherein the one or more predetermined 3D models are associated with respective model identifiers, and wherein the one or more processors of the computing device being configured to determine the identifier associated with the object or the person comprises the one or more processors being configured to: determine a classification for the object or the person based on the one or more images; match the classification to a candidate model stored in a database; and set the model identifier of the candidate model as the identifier of the object or the person.
2. 2 . The system of claim 1 , wherein the at least one processor of the visualization device being configured to determine the 3D model for the object or the person comprises the at least one processor of the visualization device being configured to determine a candidate 3D model for the object or the person by matching the identifier associated with the object or the person with the model identifier associated with the candidate 3D model.
3. 3 . The system of claim 2 , wherein the at least one processor of the visualization device being configured to determine the 3D model for the object or the person further comprises the at least one processor of the visualization device being configured to modify the candidate 3D model based at least on the 3D shape of the object or the person indicated by the representation information.
4. 4 . The system of claim 3 , wherein the one or more processors of the computing device are further configured to determine a 3D pose of the object or the person based on the one or more images, wherein the representation information further indicates the 3D pose of the object or the person, and wherein the at least one processor of the visualization device is configured to modify the candidate 3D model further based on the 3D pose of the object or the person.
5. 5 . The system of claim 4 , wherein the at least one processor of the visualization device being configured to determine the 3D model for the object or the person identified in the medical environment comprises the at least one processor of the visualization device being configured to determine that none of the one or more predetermined 3D models matches the object or the person and generate the 3D model for the object or the person based on the 3D shape and 3D pose of the object or the person indicated by the representation information.
6. 6 . The system of claim 1 , wherein the one or more predetermined 3D models include one or more 3D object models associated with a medical device in the medical environment, the one or more predetermined 3D models further including one or more 3D human models associated with a patient or medical personnel in the medical environment.
7. 7 . The system of claim 6 , wherein the one or more predetermined 3D models include one or more computer-aided design (CAD) models or one or more mesh models.
8. 8 . The system of claim 1 , wherein the one or more predetermined 3D models are stored in the database and wherein the database is accessible to at least one of the computing device or the visualization device.
9. 9 . The system of claim 1 , wherein the one or more processors of the computing device being configured to identify the object or the person in the medical environment comprises the one or more processors of the computing device being configured to detect the object or the person in the one or more images based on a machine learning model pre-trained for object or human detection.
10. 10 . The system of claim 1 , wherein the one or more processors of the computing device being configured to identify the object or the person in the medical environment comprises the one or more processors of the computing device being configured to determine at least the 3D shape of the object or the person based on a machine learning model pre-trained for 3D object or human shape estimation.
11. 11 . A method for visualizing a medical environment, the method comprising: obtaining, at a computing device, one or more images of the medical environment; identifying, at the computing device, an object or a person in the medical environment based on the one or more images, wherein the identifying comprises determining at least an identifier associated with the object or the person, a position of the object or the person in the medical environment, and a three-dimensional (3D) shape of the object or the person; generating, at the computing device, representation information that indicates at least the identifier associated with the object or the person, the position of the object or the person in the medical environment, and the 3D shape of the object or the person; obtaining, at a visualization device, the representation information generated by the computing device; determining, at the visualization device, a 3D model for the object or the person identified in the medical environment based on the representation information obtained from the computing device and one or more predetermined 3D models; and generating, at the visualization device, a visual depiction of at least the object or the person in the medical environment based on the determined 3D model and the position of the object or the person indicated by the representation information; wherein the one or more predetermined 3D models are associated with respective model identifiers, and wherein the determining the identifier associated with the object or the person comprises: determining a classification for the object or the person based on the one or more images; matching the classification to a candidate model stored in a database; and setting the model identifier of the candidate model as the identifier of the object or the person.
12. 12 . The method of claim 11 , wherein determining the 3D model for the object or the person comprises: determining a candidate 3D model for the object or the person by matching the identifier associated with the object or the person with the model identifier associated with the candidate 3D model; and modifying the candidate 3D model based at least on the 3D shape of the object or the person indicated by the representation information to obtain the 3D model for the object or the person.
13. 13 . The method of claim 12 , further comprising determining, at the computing device, a 3D pose of the object or the person based on the one or more images, wherein the representation information further indicates the 3D pose of the object or the person, and where the 3D model for the object or the person is obtained by modifying the candidate 3D model further based on the 3D pose of the object or the person.
14. 14 . The method of claim 13 , wherein determining the 3D model for the object or the person identified in the medical environment comprises determining that none of the one or more predetermined 3D models is suitable for the object or the person, and generating the 3D model for the object or the person based at least on the 3D shape and the 3D pose of the object or the person indicated by the representation information.
15. 15 . The method of claim 11 , wherein the one or more predetermined 3D models include one or more computer-aided design (CAD) object models or mesh object models associated with a medical device in medical environment, the one or more predetermined 3D models further including one or more CAD human models or mesh human models associated with a patient or medical personnel in the medical environment.
16. 16 . The method of claim 11 , wherein identifying the object or the person in the medical environment comprises detecting the object or the person in the one or more images based on a machine learning model pre-trained for object or human detection.
17. 17 . The method of claim 11 , wherein identifying the object or the person in the medical environment comprises determining at least the 3D shape of the object or the person based on a machine learning model pre-trained for 3D object or human model estimation.

Description

BACKGROUND The popularity of visualization devices such as virtual reality (VR) devices and augmented reality (AR) devices together with the creation of a shared, immersive, persistent, three-dimensional (3D) virtual space (e.g., a metaverse) may greatly enhance the ways in which humans can interact with each other in many areas of life. For example, in the healthcare field, doctors can use the metaverse and VR/AR devices to create 3D clinical applications, to make a patient's experience more interactive, to monitor a patient's vital signs in a virtual environment, etc. Complex healthcare activities, such as a medical operation requiring surgery, may be aided by a virtual representation of an operating room that is very realistic (e.g., in a VR/AR space). A high degree of realism in a virtual environment may be achieved by streaming the environment information in real time to a visualization application (e.g., the app may be associated with a pair of VR glasses or any other connected computing device) in order to give the operating doctors and technicians the confidence and feeling of completeness while using the visualization application for surgical planning or even during the surgery. However, trying to stream a large amount of 3D data from multiple sensing devices, such as cameras, that may be installed in a medical environment to the visualization application may create a bandwidth bottleneck even with the use of data compression methods and high speed communication networks (e.g., 5G networks) to reduce the bandwidth requirements. SUMMARY Described herein are systems, methods and instrumentalities associated with generating a visualization of an object or a person within a medical environment based on predetermined 3D models. A system as described herein may comprise a computing device configured to obtain one or more images of a medical environment (e.g., from sensing devices in the medical environment as described below) and identify, based on the one or more images, an object or a person in the medical environment, wherein, as a part of the identification, the computing device may be configured to determine at least an identifier associated with the object or the person, a position of the object or the person in the medical environment, and a three-dimensional (3D) shape of the object or the person. The computing device may further be configured to generate representation information that indicates at least the identifier associated with the object or the person, the position of the object or the person in the medical environment, and the 3D shape of the object or the person. The system may also include a visualization device (e.g., a virtual reality headset or a personal computing device) configured to obtain the representation information generated by the computing device and determine, based on the representation information obtained from the computing device and one or more predetermined 3D models, a 3D model for the object or the person identified in the medical environment. The visualization device may be further configured to generate a visual depiction of at least the object or the person in the medical environment based on the determined 3D model and the position of the object or the person indicated by the representation information. In some embodiments, the one or more predetermined 3D models may be associated with respective model identifiers and the visualization device may be configured to determine the 3D model for the object or the person based on determining a candidate 3D model for the object or the person by matching the identifier associated with the object or the person with the model identifier associated with the candidate 3D model. In some embodiments, the visualization device may be configured to determine the 3D model for the object or the person by modifying the candidate 3D model based at least on the 3D shape of the object or the person indicated by the representation information. In some embodiments, the computing device may be further configured to determine a 3D pose of the object or the person based on the one or more images, wherein the representation information further indicates the 3D pose of the object or the person, and wherein the visualization device is configured to modify the candidate 3D model further based on the 3D pose of the object or the person to obtain the 3D model for the object or the person. In some embodiments, the visualization device may be configured to determine that none of the one or more predetermined 3D models matches the object or the person and to generate the 3D model for the object or the person based on the 3D shape and 3D pose of the object or the person indicated by the representation information. In some embodiments, the one or more predetermined 3D models may include one or more 3D object models associated with a medical device in the medical environment and the one or more predetermined 3D models may further include one or mor