Search

KR-20260062993-A - USING OPTICAL CODES WITH AUGMENTED REALITY DISPLAYS

KR20260062993AKR 20260062993 AKR20260062993 AKR 20260062993AKR-20260062993-A

Abstract

A technique for using a medical device or a projection image by referencing an image data set and the human body is described. The method of the invention may include the step of detecting visual image data of a patient's body and a medical device. Optical codes on the patient's body and the medical device may be identified. One task is to align the image data set with the human body using one or more optical codes on the human body and fixed positions of image visibility markers for the optical codes. The position of the medical device relative to the human body may be determined by referencing the medical device to the image data set and the human body using the medical device and one or more optical codes on the human body.

Inventors

  • 기비 웬델 알렌
  • 크베트코 스티븐 토드

Assignees

  • 노바라드 코포레이션

Dates

Publication Date
20260507
Application Date
20191115
Priority Date
20181117

Claims (20)

  1. A method of using an augmented reality (AR) display to align image projections and perspective images with respect to the human body from an image data set, A step of detecting a movable viewing device relative to the human body and visual image data of a human body part using a light sensor of the AR display; A step of identifying one or more optical codes on a human body and a viewing device - one or more optical codes on the human body have fixed positions relative to image visible markers; A step of aligning an image data set of a human body using fixed positions of image visible markers for one or more optical codes on the human body when viewed through the AR display; A step of determining the position and orientation of a viewing device relative to the human body using one or more optical codes on the viewing device; A step of generating an image projection of an image data set based partially on the position and orientation of a projection device; A step of displaying an image projection through an AR display; and A method for using an augmented reality display, comprising the step of projecting an image based on the position and orientation of a viewing device or an image visibility marker using an AR display, and projecting a viewing image from a viewing device aligned with a human body.
  2. In paragraph 1, A step of detecting changes in the position and orientation of a projection device on the human body; and A method for using an augmented reality display, further comprising the step of modifying the image projection and the position and orientation of the projection image as defined by the change in the position and orientation of the projection device.
  3. A method for using an augmented reality display, wherein the step of adjusting the position of an image projection visible through an AR display is based on a change in the position and orientation of a viewing device detected using one or more optical codes on a viewing device relative to the position and orientation of a patient's body.
  4. In paragraph 1, A step of receiving a zoom value from a viewing device zoomed in on the human body; and A method for using an augmented reality display, further comprising the step of adjusting an image data set defined by the zoom value of a projection device.
  5. In paragraph 4, the step of adjusting the image data set is performed based on using the size of an image visible marker captured in the perspective image to match the zoom of the perspective image.
  6. A method of using an augmented reality display in which an object visible through a perspective image can be shown and guided by a medical professional in relation to an image data set aligned with the human body, in accordance with claim 1.
  7. A method for using an augmented reality display according to claim 1, further comprising the step of providing a graphic indicator, a virtual tool, or a virtual targeting system on an image data set to guide the position and orientation of an object that is projected onto the human body and the image data set when viewed using the AR display.
  8. A method for using an augmented reality display according to claim 1, wherein the step of determining the orientation of the viewing device further includes the step of determining the position and orientation of the viewing device relative to the human body.
  9. A method for using an augmented reality display according to claim 1, further comprising the step of reconstructing the image projection of an image data set and moving the projection image when viewed on an AR display in response to changes in the orientation and position of the projection device.
  10. A method for using an augmented reality display in which the transparency of a perspective image aligned with an image data set can be modified, in accordance with claim 1.
  11. In paragraph 1, A step of identifying one or more additional optical codes visibly displayed on a medical device; and A method for using an augmented reality display, further comprising the step of determining image projection and the position of the medical device relative to the human body using one or more additional optical codes on the medical device and one or more optical codes on the human body so that the medical device can refer to image data when viewed through the AR display.
  12. A method of using an augmented reality (AR) display to align a vision image to a human body having one or more optical codes by using the position and orientation of a vision device having one or more optical codes. A step of identifying one or more optical codes on a human body and a viewing device movable relative to the human body using a light sensor of an AR display; A step of determining the position and orientation of a viewing device relative to the human body using one or more optical codes on the viewing device; and A method for using an augmented reality display, comprising the step of displaying a fluorescent see-through image of a see-through device aligned with a human body using an AR display by referring to an optical code on a human body and the position and orientation of the see-through device.
  13. In Paragraph 12, A step of identifying one or more optical codes on a human body having a fixed position relative to an image visible marker; A step of aligning an image dataset of a human body using fixed positions of image visibility markers for one or more optical codes on the human body when viewed through an AR display; A method for using an augmented reality display, further comprising the step of displaying an image data set through an AR display in a state aligned with a human body and a perspective image.
  14. In Paragraph 13, A step of generating an image projection of an image data set based on the position and orientation of a projection device; and A method for using an augmented reality display, further comprising the step of displaying an image projection through an AR display aligned with the human body, along with a perspective image.
  15. In paragraph 13, a method of using an augmented reality display in which the image projection can be reconstructed based on the movement of the projection device to correspond to the changed position of the projection device.
  16. A method for using an augmented reality display according to claim 12, further comprising the step of generating an image projection of an image data set based on the position and orientation of a viewer using an AR headset.
  17. As a method of using an augmented reality (AR) display to align ultrasound images with the human body, A step of detecting image data of an ultrasonic transducer and a human body part using a light sensor of the AR display; A step of identifying one or more optical codes on the human body and an ultrasonic transducer; A step of determining the position and orientation of the ultrasonic transducer relative to the human body using one or more optical codes on the ultrasonic transducer; and A method for using an augmented reality display, comprising the step of displaying an ultrasonic image of an ultrasonic transducer aligned with a human body by referencing the position and orientation of an optical code on a human body and an ultrasonic transducer using the AR display.
  18. As a method of verifying medical procedures using optical codes, A step of detecting visual image data of a patient's body part and a medical device using a light sensor of an AR headset; A step of identifying one or more optical codes visibly displayed on a patient's body and a medical device—one or more optical codes on the patient's body are in a fixed position relative to an image visibility marker; A step of aligning an image data set with the patient's body using known fixed positions of image visible markers by referring to one or more optical codes on the patient's body; A step of confirming that the correct patient is in the medical procedure based on the accurate alignment of the patient's body surface aligned with the surface of the image data set above; and A method for verifying a medical procedure, comprising the step of using one or more optical codes to verify that the correct part of the body and the correct medical tool are in the medical procedure.
  19. A method for verifying a medical procedure according to claim 18, further comprising the step of loading patient data to be displayed on an AR headset using one or more optical codes.
  20. A method for verifying a medical procedure according to claim 18, further comprising the step of providing annotations on an image dataset to guide the position and orientation of a medical device relative to a patient's body and an image dataset using an AR headset.

Description

Augmented Reality Displays Using Optical Codes Mixed or augmented reality is a field of computing technology where images from the physical world and the virtual computing world can be combined into a mixed reality world. In mixed reality, the environment becomes one where people, places, and objects from both the physical and virtual worlds are blended. Mixed reality experiences can be provided through existing commercial or custom software, along with the use of VR (Virtual Reality) or AR (Augmented Reality) headsets. Augmented Reality (AR) is an example of mixed reality where a live, direct, or indirect view of a real-world environment is augmented or supplemented by computer-generated sensory inputs such as sound, video, graphics, or other data. Augmentation is performed by viewing the actual location in relation to environmental elements. With the help of advanced AR technologies (e.g., the addition of computer vision and object recognition), information about the user's surrounding real world can become interactive and digitally modified. The challenge facing AR systems or headsets is identifying the position and orientation of objects with high precision. Similarly, aligning the position of virtual elements with a live view of the real environment can be a challenge. While the alignment resolution of AR headsets can align virtual objects with the physical objects being viewed, the alignment is limited to within a few centimeters. Providing alignment within a few centimeters may be useful for entertainment and less demanding applications, but much higher positioning and alignment resolutions for AR systems may be required in scientific, engineering, and medical fields. Consequently, the positioning and alignment process is performed manually, which can be time-consuming, cumbersome, and inaccurate. FIG. 1 illustrates an exemplary augmented reality (AR) environment in which image data of a medical device and a patient can be referenced and aligned with a real view of the patient using one or more optical codes attached to the patient and the medical device. Figure 2 illustrates an example of an optical code having an image visible marker fixed to a patient. Figure 3 illustrates an example of visual data that can be captured by a real-view camera of a patient's body and medical device with an optical code fixed thereto. Figure 4 illustrates an example of a view in an augmented reality (AR) display with annotations to guide the positioning and orientation of a medical device during a medical procedure. FIG. 5 illustrates an example of a screen that can be displayed on an augmented reality display device to display information on the use or guidance of a medical device during a medical procedure. Figure 6 illustrates an example of using an augmented reality (AR) display to enable perspective images and image projections from an image data set of the human body using an optical code. FIG. 7 illustrates an example of a movable fluorescent fluoroscopy device for the human body, and the fluoroscopy image and image projection are also moved and/or modified. FIG. 8a illustrates a fluorescent viewing device movable over a human body that generates a viewing image and image projection in a coronal view of a viewing device that enables combined viewing through an augmented reality (AR) headset or AR display. FIG. 8b illustrates an ultrasonic transducer used in combination with an optical code to enable a combined view of an image data set and an ultrasonic image through an augmented reality (AR) headset or AR display. Figure 9a is a flowchart of an exemplary method of using an augmented reality headset to reference an image data set and a medical device on a patient's body. FIG. 9b illustrates a method of using an optical code to align a fluorescent image, an image projection from an image data set, and an augmented reality (AR) display. FIG. 10 illustrates an exemplary system that can be used to use an optical code on a medical device to enable the medical device to be referenced to an image data set and the patient's body. FIG. 11 is a block diagram showing an example of a computing system that processes the present technology. Technology is provided for using an augmented reality (AR) headset to enable one or more optical codes to be identified on a medical device from the perspective of the AR headset's camera during a medical procedure. The medical device may refer to an image dataset aligned with the human body using one or more optical codes and image-visible markers located on the human body. The image dataset may be a portion of the human body image acquired in advance using non-optical imaging techniques (e.g., MRI (Magnetic Resonance Imaging), CT (Computed Tomography) scanning, X-ray, etc.). The image dataset may be aligned with the human body using an image-visible marker that is at a fixed distance from at least one optical code located on the human body. For example, both the image-visible marker and