CN-116542918-B - Image registration model training method and device and image processing method

Abstract

The invention provides an image registration model training method, which relates to the technical field of image processing and comprises the steps of utilizing a deformation learning sub-model to process a fundus image sample to be registered to obtain a deformation image sample, obtaining deformation image key point information and reference image key point information, obtaining a first loss value based on the deformation image key point information and the reference image key point information, adjusting parameters of the deformation learning sub-model based on the first loss value, utilizing a second loss value to adjust an inverse deformation learning sub-model, cycling the iteration deformation learning sub-model until the first loss value meets a first convergence condition and the second loss value meets a second convergence condition, obtaining a trained deformation learning sub-model, and determining the trained deformation learning sub-model as an image registration model. The image registration model obtained by the image registration model training method can register images of different modes, and the purpose of improving the precision of the multi-mode fundus image registration result is achieved.

Inventors

• ZHOU DENGJI
• Ling Saiguang

Assignees

• 依未科技(北京)有限公司

Dates

Publication Date

20260505

Application Date

20230424

Claims (10)

1. 1. An image registration model training method, comprising: Processing the fundus image sample to be registered based on the fundus image sample to be registered and a reference image sample by utilizing a deformation learning sub-model to obtain a deformation image sample, wherein the mode of the fundus image sample to be registered is different from the mode of the reference image sample; extracting key points of the deformed image sample by using a first characteristic point extraction sub-model to obtain deformed image key point information; Extracting key points of the reference image sample by using the first characteristic point extraction sub-model to obtain reference image key point information; Based on the deformation image key point information and the reference image key point information, performing loss calculation by using a first loss function to obtain a first loss value; Based on the first loss value, adjusting parameters of the deformation learning sub-model; Performing inverse deformation processing on the reference image sample based on the fundus image sample to be registered and the reference image sample by using an inverse deformation learning submodel to obtain an inverse deformation reference image sample; obtaining a second loss value based on the inverse deformation reference image sample and the fundus image sample to be registered; Adjusting parameters of the inverse deformation learning sub-model by using the second loss value, and circularly iterating the inverse deformation learning sub-model; And iterating the deformation learning sub-model circularly until the first loss value meets a first convergence condition and the second loss value meets a second convergence condition, obtaining a trained deformation learning sub-model, determining the trained deformation learning sub-model as an image registration model, wherein the image registration model is used for deforming the fundus image to be registered based on the fundus image to be registered and a reference image to obtain a deformed image, and determining the deformed image as a registration image corresponding to the fundus image to be registered.
2. 2. The method of claim 1, wherein the deformation learning sub-model comprises a first convolutional neural network layer, a second convolutional neural network layer, a third convolutional neural network layer, a first transducer layer, and a second transducer layer, wherein the deformation processing comprises a first deformation processing and a second deformation processing, wherein the processing the fundus image sample to be registered based on the fundus image sample to be registered and a reference image sample using the deformation learning sub-model to obtain a deformed image sample comprises: determining a first deformation field based on the fundus image sample to be registered and the reference image sample by using the first convolutional neural network layer; Extracting local features of the fundus image sample to be registered by using the second convolutional neural network layer; Performing the first deformation processing on the local features of the fundus image sample to be registered based on the first deformation field by using the third convolutional neural network layer, and determining a local feature deformation result of the fundus image sample to be registered; Extracting global features of the fundus image sample to be registered based on the fundus image sample to be registered by using the first transducer layer; and carrying out second deformation processing by using the second transducer layer based on the first deformation field, the local characteristic deformation result of the fundus image sample to be registered and the global characteristic of the fundus image sample to be registered, so as to obtain the deformed image sample.
3. 3. The method according to claim 1, wherein said obtaining a second loss value based on said inverse transformed reference image sample and said fundus image sample to be registered comprises: Extracting key points of the inverse deformation reference image sample by using a second characteristic point extraction sub-model to obtain inverse deformation reference image key point information; Extracting key points of the fundus image sample to be registered by using the second characteristic point extraction sub-model to obtain fundus image key point information to be registered; And carrying out loss calculation by using a second loss function based on the inverse deformation reference image key point information and the fundus image key point information to be registered to obtain a second loss value.
4. 4. A method according to claim 3, further comprising: Utilizing an inverse deformation learning sub-model, based on the deformation image sample and the fundus image sample to be registered, performing inverse deformation treatment on the deformation image sample to obtain an inverse deformation image sample; based on the inverse deformation image sample and the fundus image sample to be registered, performing loss calculation by using a third loss function to obtain a third loss value; based on the third loss value, adjusting parameters of the deformation learning sub-model; The loop iterates the deformation learning sub-model until the first loss value meets a first convergence condition and the second loss value meets a second convergence condition, and a trained deformation learning sub-model is obtained, comprising: And circularly iterating the deformation learning sub-model until the first loss value meets the first convergence condition, the second loss value meets the second convergence condition, and the third loss value meets the third convergence condition, so as to obtain the trained deformation learning sub-model.
5. 5. The method as recited in claim 4, further comprising: performing deformation processing on the inverse deformation reference image sample based on the inverse deformation reference image sample and the reference image sample by using the deformation learning sub-model to obtain a deformation reference image sample; based on the deformed reference image sample and the reference image sample, performing loss calculation by using a fourth loss function to obtain a fourth loss value; based on the fourth loss value, adjusting parameters of the deformation learning sub-model; The loop iterates the deformation learning sub-model until the first loss value meets the first convergence condition, the second loss value meets the second convergence condition, and the third loss value meets the third convergence condition, and the trained deformation learning sub-model is obtained, comprising: And circularly iterating the deformation learning sub-model until the first loss value meets the first convergence condition, the second loss value meets the second convergence condition, the third loss value meets the third convergence condition, and the fourth loss value meets the fourth convergence condition, so as to obtain the trained deformation learning sub-model.
6. 6. The method according to any one of claims 1 to 5, characterized in that before said processing of the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample with the deformation learning submodel, further comprises: Based on an initial reference image sample and the fundus image sample to be registered, cutting the initial reference image sample by using an area positioning sub-model to obtain the reference image sample, wherein the shooting view of the initial reference image sample is larger than that of the fundus image sample to be registered, and the fundus image area included in the reference image sample is consistent with the fundus image area view included in the fundus image sample to be registered.
7. 7. An image processing method, comprising: Acquiring a plurality of fundus images of different modes; Determining one fundus image of the fundus images with different modes as a reference image, and determining fundus images except the reference image in the fundus images with different modes as at least one fundus image to be registered; And registering the at least one fundus image to be registered by utilizing an image registration model based on the reference image and the at least one fundus image to be registered to obtain respective registration images of the at least one fundus image to be registered, wherein the image registration model is determined based on the image registration model training method of any one of 1 to 6.
8. 8. An image registration model training apparatus, comprising: The deformation module is used for processing the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample by utilizing the deformation learning submodel to obtain a deformation image sample, and the mode of the fundus image sample to be registered is different from the mode of the reference image sample; the first feature extraction module is used for extracting key points of the deformed image sample by utilizing the first feature point extraction sub-model to obtain deformed image key point information; the first feature extraction module is further used for extracting key points of the reference image sample by using the first feature point extraction sub-model to obtain reference image key point information; the loss calculation module is used for carrying out loss calculation by utilizing a first loss function based on the deformation image key point information and the reference image key point information to obtain a first loss value; The adjustment module is used for adjusting parameters of the deformation learning sub-model based on the first loss value; The inverse deformation processing module is used for carrying out inverse deformation processing on the reference image sample based on the fundus image sample to be registered and the reference image sample by utilizing an inverse deformation learning submodel to obtain an inverse deformation reference image sample; the loss calculation module is further used for obtaining a second loss value based on the inverse deformation reference image sample and the fundus image sample to be registered; the adjusting module is further configured to adjust parameters of the inverse deformation learning sub-model by using the second loss value, and iterate the inverse deformation learning sub-model in a cyclic manner; The determining module is used for circularly iterating the deformation learning sub-model until the first loss value meets a first convergence condition and the second loss value meets a second convergence condition, obtaining a trained deformation learning sub-model, determining the trained deformation learning sub-model as an image registration model, wherein the image registration model is used for deforming the fundus image to be registered based on the fundus image to be registered and a reference image to obtain a deformed image, and determining the deformed image as a registration image corresponding to the fundus image to be registered.
9. 9. An electronic device, comprising: A processor; a memory for storing the processor-executable instructions, Wherein the processor is adapted to perform the method of any of the preceding claims 1 to 7.
10. 10. A computer readable storage medium, characterized in that the storage medium stores a computer program for executing the method of any of the preceding claims 1 to 7.

Description

Image registration model training method and device and image processing method Technical Field The disclosure belongs to the technical field of image processing, and particularly relates to an image registration model training method and device and an image processing method. Background Medical fundus images have been widely used to record and examine clinical manifestations of a variety of diseases, fundus image registration being one of the fundamental means of fundus image processing and analysis. The fundus image registration can spatially align two or more fundus images to assist a doctor in performing ophthalmic diagnosis and treatment, and thus has important clinical application value. Because the single-mode fundus image information is single, the common fundus camera image, the fundus fluorescence angiography image, the optical coherence tomography angiography (Optical Coherence Tomography Angiography, OCTA) image and other fundus images in different modes are generally comprehensively analyzed in clinical application, however, the traditional image registration method cannot be adapted to multi-mode fundus image registration, so that the precision of registration results of the multi-mode fundus images is low, and the development of the fundus images in clinical application is greatly limited. Disclosure of Invention In view of the above, the present disclosure provides an image registration model training method and apparatus, an image processing method, a computer readable storage medium, and an electronic device, so as to solve the problem that the accuracy of registration results of multi-modal fundus images is low because the conventional fundus image registration method cannot be adapted to multi-modal fundus image registration. In a first aspect, an embodiment of the present disclosure provides an image registration model training method, including processing a fundus image sample to be registered based on the fundus image sample to be registered and a reference image sample using a deformation learning sub-model to obtain a deformed image sample, the modality of the fundus image sample to be registered is different from that of the reference image sample, extracting key points of the deformed image sample using a first feature point extraction sub-model to obtain deformed image key point information, extracting key points of the reference image sample using a first feature point extraction sub-model to obtain reference image key point information, performing a loss calculation using a first loss function based on the deformed image key point information and the reference image key point information to obtain a first loss value, adjusting parameters of the deformation learning sub-model based on the first loss value, performing an inverse deformation processing on the reference image sample based on the fundus image sample to be registered using an inverse deformation learning sub-model to obtain an inverse deformation reference image sample, obtaining a second loss value based on the inverse deformation reference image sample and the reference image sample, adjusting parameters of the deformation learning sub-model based on the first feature point extraction sub-model, and performing an iterative learning sub-model until the first loss value is satisfied and the first loss value is satisfied, performing an iterative learning sub-model is satisfied, and the fundus image registration is performed based on the first loss value, and the first model is better, and the fundus image is registered, and determining the deformation image as a registration image corresponding to the fundus image to be registered. With reference to the first aspect, in some implementations of the first aspect, the deformation learning sub-model includes a first convolutional neural network layer, a second convolutional neural network layer, a third convolutional neural network layer, a first transducer layer and a second transducer layer, the deformation processing includes a first deformation processing and a second deformation processing, the deformation learning sub-model is utilized to process the fundus image sample to be registered and the reference image sample based on the fundus image sample to be registered, the deformation image sample is obtained, the deformation learning sub-model includes utilizing the first convolutional neural network layer to determine a first deformation field based on the fundus image sample to be registered and the reference image sample, utilizing the second convolutional neural network layer to extract local features of the fundus image sample to be registered, utilizing the third convolutional neural network layer to perform the first deformation processing based on the local features of the fundus image sample to be registered to determine a local feature deformation result of the fundus image sample to be registered, the first transducer layer is utilized to extract global fe