US-20260127222-A1 - MEDICAL IMAGE SEARCH AND RETRIEVAL

US20260127222A1US 20260127222 A1US20260127222 A1US 20260127222A1US-20260127222-A1

Abstract

A method for retrieving relevant images from an image database based on converting linguistic search criteria into image-based search criteria. User inputs are used to construct a base image from a combination of pre-formed graphical or visual image elements, and an image database is then queried with the base image using image-based searching. The resulting retrieved images may be exported to a user training system for use in training a user using the images.

Inventors

Shreya Anand
Biswaroop Chakrabarti
Sam Martin JELFS
Jose Luis Diaz MENDOZA
Raymond Van Ee

Assignees

KONINKLIJKE PHILIPS N.V.

Dates

Publication Date: 20260507
Application Date: 20231006
Priority Date: 20221011

Claims (15)

1 . A computer-implemented method comprising: selecting a plurality of pre-formed image elements from a pre-defined set of pre-formed image elements, each image element being an image representation of an anatomical object or feature; defining a relative positioning of the selected plurality of image elements relative to one another and/or a size of the selected plurality of image elements; wherein the selection of the plurality of image elements from the set, and the defining of the positioning and/or size of the image elements is performed based on user inputs received from a user interface device; constructing an artificial base image based on combining the selected image elements in accordance with the defined positioning and/or size, accessing an image database storing a dataset of medical images; performing an image-based searching operation comprising searching the image database based on the base image for finding images in the database similar to the base image; generating a data package for export to a user training system comprising at least a subset of any images identified by the image-based search.
2 . The method of claim 1 , wherein the received user inputs include at least: an input indicative of a desired anatomical structure, and an input indicative of a desired pathology associated with the anatomical structure; wherein the selection of the plurality of pre-formed image elements comprises selecting at least one image element representative of the anatomical structure and at least one image element representative of an anatomical feature associated with the pathology.
3 . The method of claim 1 , wherein the received user inputs further comprise an indication of a desired imaging modality for the base image.
4 . The method of claim 1 , wherein the received user inputs further comprise an indication of a desired 2D/3D dimensionality of the image.
5 . The method of claim 1 , wherein the method comprises providing via a display of the user interface device a graphical user interface permitting input by the user of the said user inputs.
6 . The method of claim 5 , wherein the graphical user interface comprises a set of input fields permitting input by the user of the desired anatomical structure and desired pathology.
7 . The method of claim 5 , wherein the method comprises constructing the base image in real time with receipt of each user input, and wherein the graphical user interface includes a preview pane showing the base image in a current form, and updating the base image in the preview pane following each further user input.
8 . The method of claim 5 , wherein the method comprises defining at least a relative positioning of the selected plurality of image elements, and wherein the graphical user interface provides control functionality permitting drag-and-drop by a user of the selected image elements relative to one another, for thereby indicating the relative positioning of the image elements relative to one another.
9 . The method of claim 8 , wherein the combining of the image elements to form the base images comprises overlaying at least one of the selected image elements atop at least one other of the selected image elements such that the base image has a layered formation.
10 . The method of claim 1 , wherein the method further comprises: responsive to determining that the number of images identified by the image-based search falls below a threshold number, accessing from a datastore a generative adversarial network (GAN) which is configured to receive as input the base image and to generate as output one or more simulated images; supplying the base image to the GAN to generate one or more simulated images; and including the generated one or more simulated images in the data package for export.
11 . The method of claim 10 , wherein the method further comprises comparing each of the one or more simulated images output from the GAN network against the base image; presenting on the user interface an output indicative of a result of the comparison for each image; and receiving a user input from the user interface indicative, for each of the simulated images, of user acceptance of the image or user rejection of the image; wherein only images which are accepted by the user are included in said data package for export.
12 . The method of claim 11 , wherein, in advance of performing the said comparison, an anatomical plausibility check is performed comprising processing each generated image output from the GAN with one or more algorithms, each algorithm configured to detect one or more features in the image and compare the features against one or more rules to determine an anatomical plausibility of the feature.
13 . The method of claim 1 , further comprising: receiving a training results report from the user training system, the results report indicative of a level of user training success in relation to one or more anatomical structures and/or pathologies of anatomical structures; determine based on the training results report one or more anatomical structures and/or pathologies of anatomical structures for which a level of user success is below a pre-defined threshold; wherein the selection of the plurality of image elements from the set is performed based on the said determined one or more anatomical structures and/or pathologies of anatomical structures.
14 . A computer program product comprising code means configured when run on a processor to cause the processor to perform a method in accordance with claim 1 .
15 . A processing unit, comprising: an input/output; and one or more processors configured to perform a method, wherein the method comprises: receiving at the input/output a set of user inputs from a user interface device; selecting a plurality of pre-formed image elements from a pre-defined set of pre-formed image elements, each image element being an image representation of an anatomical object or feature; defining a relative positioning of the selected plurality of image elements relative to one another and/or a size of the selected plurality of image elements; wherein the selection of the plurality of image elements from the set, and the defining of the positioning and/or size of the image elements is performed based on the user inputs received from the user interface device; constructing an artificial base image based on combining the selected image elements in accordance with the defined relative positioning and/or size(s), accessing an image database storing a dataset of medical images; performing an image-based searching operation comprising searching the image database based on the base image for finding images in the database similar to the base image; generating a data package for export via the input/output to a user training system comprising at least a subset of any images identified by the image-based search.

Description

FIELD OF THE INVENTION The present invention relates to a method for optimized search of medical images having defined anatomical requirements. BACKGROUND OF THE INVENTION Advancement of technology, particularly with regards to data storage, has resulted in availability of a large amount of medical image data stored in institutional and pan-institutional PACS systems. This information has enormous potential value, in particular for use in training clinicians in diagnostic analysis, and also potentially in training AI algorithms for various diagnostic and other classification functions. However, despite the availability of the image data, there is a challenge in sorting the image data to find and retrieve images and image sets based on defined query criteria. The standard way to search a PACS database is to search by keyword and rely on metadata tags of the images in order to identify relevant images that meet the search criteria. However, tags may not always be reliable, or may not be as comprehensive as they could be. Some images may have no descriptive tags at all. Therefore, there is a need for an improved means of obtaining image data sets from image databases based on defined criteria. SUMMARY OF THE INVENTION The invention is defined by the claims. According to examples in accordance with an aspect of the invention, there is provided a computer-implemented method comprising: selecting a plurality of pre-formed image elements from a pre-defined set of pre-formed image elements, each image element being an image representation of an anatomical object or feature; defining a relative positioning and/or size of the selected one or more image elements, wherein the selection of the plurality of image elements from the set, and the defining of the positioning and/or size of the image elements is performed based on user inputs received from a user interface device; constructing an artificial base image based on combining the selected image elements in accordance with the defined positioning and/or size; accessing an image database storing a dataset of medical images; performing an image-based searching operation comprising searching the image database based on the base image for finding images in the database similar to the base image; and generating a data package for export to a user training system comprising at least a subset of any images identified by the image-based search. The inventive realization according to embodiments of this invention is to realize that it is easier and more reliable to search an image database visually, using image-based searching, than it is to use more typical methods of searching using e.g. tagged metadata of the images. Image-based searching is a much more direct, reliable, robust way of searching for what is needed. Algorithms for image based searching already exist. Therefore the inventive concept is to realize that the problem of obtaining image data in accordance with requirements can be solved by first converting linguistically represented criteria for the image (expressed for example via selection of options on an interactive UI) into a graphical representation, in the form of a constructed artificial base image, which is formed from a layered formation of different visual/graphical elements overlaid and size and position-adjusted. Each visual/graphical element is for example a pre-formed template image element representative of an anatomical object or feature. In the context of the present disclosure, ‘pre-formed’ means, for example, that each of the pre-defined set of image elements is separated or isolated in advance, so that each comprises a single graphical object for example. Each pre-formed image element may be a separate individual image in some examples. The different image elements may have different outer boundary shapes in some examples. The database can then be searched using the base image. The constructed base image itself would not be suitable for training purposes because it is an artificial construct, i.e. a rough visual seed used for searching (or for image generation using a neural network). The term ‘relative positioning’ means for example a positioning of the image elements relative to one another. The defined size of the image elements may mean a relative size relative to one another, or an absolute size, e.g. in pixel scale. As will be explained below, in accordance with some embodiments, if the database finds no or few results, then a pre-trained AI model can be used to generate new images based on the pre-existing image. In some embodiments, the received user inputs include at least: an input indicative of a desired anatomical structure, and an input indicative of a desired pathology associated with the anatomical structure. The selection of the plurality of pre-formed image elements may comprise selecting at least one image element representative of the anatomical structure and at least one image element representative of an anatomical featur