US-12620097-B2 - Technique for optimizing rendering parameters of overlays of medical images

Abstract

An overlay of an aligned medical image obtained from a medical scanner is rendered with a reference image. The reference image has a reference structure of a body. An intermediate representation of the reference structure is determined. The medical image has structures corresponding to the reference structures, and an intermediate representation of the structures in the medical image is determined. A rendering parameter is optimized by comparing the intermediate representations of the medical image and of the reference image. The medical image is aligned and overlayed with the reference image based on the optimized rendering parameter, and the aligned and overlayed image is rendered.

Inventors

• Kaloian Petkov

Assignees

• Siemens Healthineers Ag

Dates

Publication Date

20260505

Application Date

20230428

Priority Date

20220506

Claims (18)

1. 1 . A computer-implemented method for rendering of an overlay of an aligned medical image from data obtained from a medical scanner with a reference image, the method comprising: receiving the reference image, wherein the reference image comprises at least one reference structure of a body; determining an intermediate representation of the at least one reference structure in the received reference image; receiving at least one medical image from a medical scanner, wherein the at least one medical image comprises at least partially structures of a patient's body corresponding to the at least one reference structure of the received reference image; determining an intermediate representation of at least part of the structures in the received at least one medical image; receiving at least one rendering parameter for rendering the received at least one medical image; optimizing the received at least one rendering parameter for aligning and overlaying the received at least one medical image with the received reference image, wherein the optimizing comprises comparing the intermediate representation of the received at least one medical image with the intermediate representation of the received reference image; aligning and overlaying the received at least one medical image with the received reference image based on the optimized at least one rendering parameter; and rendering the aligned and overlayed at least one medical image with the at least one reference image.
2. 2 . The method of claim 1 , wherein the medical scanner comprises: a surgical camera; an ultrasound scanner; a positron emission tomography scanner; an x-ray imaging scanner; a computed tomography scanner; or a magnetic resonance tomography scanner.
3. 3 . The method of claim 1 , wherein the act of rendering comprises rendering the aligned and overlayed at least one medical image with the reference image using a device, the device comprising: a screen; a stereoscopic display; a virtual reality display; or an augmented reality headset.
4. 4 . The method of claim 1 , wherein the reference image comprises: a cryosection image data set; a synthetically generated image; or an image received from a medical scanner.
5. 5 . The method of claim 1 , wherein the intermediate representation of the at least one reference structure and/or the intermediate representation of the at least partial structures of the at least one medical image is: a segmentation; a mesh; an isosurface; or a point cloud.
6. 6 . The method of claim 1 , wherein the at least one rendering parameter comprises: a view position; a field of view; a view orientation; a depth of field; and/or a lighting.
7. 7 . The method of claim 1 , wherein at least one of the received reference image and the received at least one medical image comprises a series of non-static images.
8. 8 . The method of claim 7 , wherein the series of non-static medical images is received from the medical scanner in real-time during an interventional procedure.
9. 9 . The method of claim 1 , wherein comparing the intermediate representation of the received at least one medical image with the intermediate representation the received reference image comprises generating a function of a difference of at least one property of the intermediate representation of the received at least one medical image and of the intermediate representation of the received reference image, and wherein optimizing the at least one rendering parameter comprises determining at least one partial derivative of the generated function with respect to the at least one rendering parameter.
10. 10 . The method of claim 9 , wherein the at least one property of the intermediate representation of the received at least one medical image and of the intermediate representation of the received reference image comprises an image color per discrete image location.
11. 11 . The method of claim 10 , wherein the discrete image locations comprise pixels and/or voxels.
12. 12 . The method of claim 9 , wherein optimizing the at least one rendering parameter comprises minimizing the generated function of the difference of the at least one property of the intermediate representation of the received at least one medical image and of the intermediate representation of the received reference image.
13. 13 . The method of claim 1 , wherein receiving the reference image, determining the associated intermediate representation, receiving the at least one medical image, determining the associated intermediate representation, and receiving and optimizing the at least one rendering parameter are performed by a graphics processing unit.
14. 14 . A system for rendering of an overlay of a medical image from data obtained from a medical scanner with a reference image, the system comprising: a first interface configured to receive the reference image, wherein the reference image comprises at least one reference structure of a body; a first processing unit configured to determine an intermediate representation of the at least one reference structure in the received reference image; a second interface configured to receive the medical image from the medical scanner, wherein the medical image comprises, at least partially, structures of a patient's body corresponding to the at least one reference structure of the received reference image; a second processing unit configured to determine an intermediate representation of at least part of the structures in the received medical image; a third interface configured to receive at least one rendering parameter for rendering the received medical image; a third processing unit configured to optimize the received at least one rendering parameter for aligning and overlaying the received medical image with the received reference image, wherein the optimization comprises comparison of the intermediate representation of the received medical image with the intermediate representation of the received reference image; a fourth processing unit configured to align and overlay the received medical image with the received reference image based on the optimized at least one rendering parameter; and a fourth interface configured to output the aligned and overlayed medical image with the reference image for rendering.
15. 15 . The system of claim 14 , further comprising: the medical scanner, the medical scanner comprising a surgical camera, an ultrasound scanner, a positron emission tomography scanner, an x-ray imaging scanner, a computed tomography scanner, or a magnetic resonance tomography scanner; a renderer configured to render the aligned and overlayed medial image with the reference image; and a display configured to display the rendered medial image with the reference image, the display comprising a screen, a stereoscopic display, a virtual reality display, or an augmented reality headset; wherein the reference image comprises a cryosection image data set, a synthetically generated image, or an image received from the medical scanner or another medical scanner; wherein the intermediate representation of the at least one reference structure and/or the intermediate representation of the structures of the at least one medical image is a segmentation, a mesh, an isosurface, or a point cloud; and wherein the at least one rendering parameter comprises a view position, a field of view, a view orientation, a depth of field, and/or a lighting.
16. 16 . The system of claim 14 , wherein at least one of the received reference image and the received medical image comprises a series of non-static images received from the medical scanner in real-time during an interventional procedure.
17. 17 . The system of claim 14 , wherein the third processing unit is configured to compare the intermediate representation of the received medical image with the intermediate representation the received reference image by generation a function of a difference of at least one property of the intermediate representation of the received medical image and of the intermediate representation of the received reference image, and wherein the optimization of the at least one rendering parameter comprises determination of at least one partial derivative of the generated function with respect to the at least one rendering parameter.
18. 18 . A system for rendering of an overlay of a medical image from data obtained from a medical scanner with a reference image, the system comprising: a reference database for providing at least one reference image; a medical scanner for providing at least one medical image; a computer, wherein a first interface is configured to receive the at least one reference image from the reference database, and a second interface is configured to receive the at least one medical image from the medical scanner; a processor is configured to optimize a rendering parameter for aligning and overlaying the received at least one medical image with the received at least one reference image, wherein the optimization comprises comparison of an intermediate representation of structure in the received medical image with an intermediate representation of the structure in the received reference image, the processor configured to align and overlay the received at least one medical image with the received at least one reference image based on the optimized rendering parameter; and a renderer configured to render the aligned and overlaid at least one medical image with the at least one reference image.

Description

RELATED APPLICATION This application claims the benefit of EP 22171999.0, filed May 6, 2022, which is hereby incorporated by reference in its entirety. FIELD A technique for optimizing rendering parameters of overlays of medical images over reference images is provided. In particular, a method for optimizing rendering parameters in differentiable rendering of medical images is provided. BACKGROUND Differentiable rendering (DR) models the explicit relationship between rendering parameters and resulting images in traditional image synthesis. In DR, image-space derivatives with respect to the rendering parameters are obtained, which derivatives can be used in a variety of gradient-based optimization methods to solve inverse rendering problems or to compute the loss for training machine learning models directly in the space of rendered images. While many conventional differentiable renderers, such as OpenDR, are limited to simple surface rendering and local shading, there are examples for photorealistic surface and participating media rendering. However, DR is conventionally not applied to heterogeneous volume data, which typically are obtained by medical scanners. More recently, a system for Differentiable Direct Volume Rendering was introduced. The system presents a more general approach to differentiable direct volume rendering that can solve a wider range of inverse rendering problems. It comes, however, at the cost of both computational complexity and long computing time. Conventional volume visualization methods based on ray casting, which are still used in many current advanced visualization medical products, simulate only the emission and absorption of radiant energy along the primary viewing rays through the volume data. The emitted radiant energy at each point is absorbed according to the Beer-Lambert law along the ray to the observer location with absorption coefficients derived from the patient data. Renderers conventionally compute shading using only standard local shading models at each point along the ray (e.g., the Blinn-Phong model), based on local volume gradients (i.e., local illumination). While fast, these methods do not simulate the complex light scattering and extinction associated with photorealism (i.e., global illumination). Physically-based Monte Carlo light transport simulates light paths though the volume data with multiple scattering events per path using a stochastic process. As more and more paths are simulated, the solution converges on an accurate estimation of the irradiance at each point for incoming light from all directions. The renderer employs a hybrid of volumetric scattering and surface-like scattering, modeled by phase functions and bidirectional reflectance distribution functions (BRDFs), respectively, based on properties derived from the anatomical data. The conventional techniques are, however, limited by both computational speed and computational memory, in particular for heterogenous volume data, rendering them unsuitable for overlays of medical images of heterogeneous volume data, in particular at real-time. SUMMARY AND DESCRIPTION It is therefore an object to provide an efficient solution for a photorealistic overlay of medical heterogenous volume data, including a medical image and a reference image that are heterogenous. Alternatively, or additionally, there is a need for a photorealistic technique for providing photorealistic overlays, in particular of heterogeneous medical image data, that is efficient in time and/or computational effort. This object is solved by a computer implemented method for rendering of an overlay of an aligned medical image from data obtained from a medical scanner (which may also be denoted as medical imaging device or medical image scanner) with a reference image, by a computing device, by a system including the medical scanner and the computing device, by a computer program and/or computer program product, and by a non-transitory computer-readable medium. Advantageous aspects, features and embodiments are described in the following description together with advantages. In the following, the solution is described with respect to the method for rendering of an overlay of an aligned medical image from data obtained from a medical scanner with a reference image as well as with respect to the computing device and the system including the computing device. Features, advantages, and/or alternative embodiments herein can be assigned to the other claimed objects (e.g., the computer program or a computer program product), and vice versa. In other words, claims for the computing device and/or for the system can be improved with features described or claimed in the context of the method. In this case, the functional features of the method are embodied by structural units or devices of the computing device and/or of the system and vice versa, respectively. As to a first aspect, a computer-implemented method for rendering of an overlay of an ali