Search

CN-122005091-A - Machine vision-based ovarian cancer intraoperative navigation method and system

CN122005091ACN 122005091 ACN122005091 ACN 122005091ACN-122005091-A

Abstract

The invention discloses an ovarian cancer intraoperative navigation method and system based on machine vision, comprising the following steps of firstly obtaining a fluorescence signal, secondly obtaining a white light image and a fluorescence image, thirdly obtaining a fluorescence correction image and a white light reliable region mask, fourthly inputting the fluorescence correction image into an improved SAM2 model for tumor segmentation and outputting a tumor region mask and a tumor confidence map, fifth generating a white light structure map and a fluorescence structure map, sixth extracting a white light key point set and a fluorescence key point set, seventh performing key point matching by utilizing a bidirectional consistency constraint rule of a SuperGlue algorithm to generate a registration mapping relation from the fluorescence image to the white light image, eighth mapping the tumor region mask and the tumor confidence map to a white light image coordinate system, and generating an intraoperative navigation image. The invention combines the improved SAM2 model and the SuperGlue algorithm to realize accurate navigation in the ovarian cancer operation.

Inventors

  • LIU BING
  • QU JING
  • PAN YUE
  • LIU LIFENG
  • WANG XIAOFENG
  • CUI YUEMEI
  • LI ZHENGYAN
  • YU HAN

Assignees

  • 大连理工大学附属中心医院(大连市中心医院)

Dates

Publication Date
20260512
Application Date
20260214

Claims (10)

  1. 1. An ovarian cancer intraoperative navigation method based on machine vision is characterized by comprising the following steps: injecting a tumor microenvironment response type NIR-IIb composite nano probe into a subject to be operated before ovarian cancer operation to obtain a fluorescence signal; synchronously acquiring a white light image and a fluorescence image corresponding to the fluorescence signal at the same time point in the ovarian cancer operation process to obtain a white light image and a fluorescence image; step three, carrying out normalization processing and discrete wavelet transformation on the fluorescent image to obtain a fluorescent correction image, and carrying out reliable region identification on the white light image to obtain a white light reliable region mask; Inputting the fluorescence correction image into an improved SAM2 model for tumor segmentation, wherein the improved SAM2 model comprises an encoder, a hierarchical memory stack, a bootstrap prompting ring and a two-domain representation decoder which are connected in sequence, and outputting a tumor region mask and a tumor confidence map; Generating a white light structure chart based on the white light image and the white light reliable region mask, and generating a fluorescence structure chart based on the fluorescence correction image, the tumor region mask and the tumor confidence chart; Step six, extracting a white light key point set based on the white light structure diagram, and extracting a fluorescent key point set based on the fluorescent structure diagram; Step seven, performing key point matching by utilizing a bidirectional consistency constraint rule of a SuperGlue algorithm based on a white light key point set and a fluorescent key point set to generate a registration mapping relation from a fluorescent image to a white light image; And step eight, mapping the tumor region mask and the tumor confidence map to a white light image coordinate system based on a registration mapping relation from the fluorescent image to the white light image, and generating an intraoperative navigation image.
  2. 2. The machine vision-based intra-operative navigation method for ovarian cancer as set forth in claim 1, wherein the first step is specifically: injecting a tumor microenvironment response type NIR-IIb composite nano probe into a subject to be operated before ovarian cancer operation; the NIR-IIb composite nano probe circulates in the body and is enriched in the tissue region of ovarian cancer tumor; Under the action of a microenvironment that the tumor tissue region presents glutathione enrichment relative to the normal tissue region, the structure change and the energy level state change of the NIR-IIb composite nano probe are generated, so that the NIR-IIb composite nano probe recovers fluorescence emission capability in the tumor tissue region and emits a fluorescence signal with the wavelength of 1650 nm.
  3. 3. The machine vision-based ovarian cancer intraoperative navigation method of claim 1, wherein the second step is specifically: in the ovarian cancer operation process, a white light imaging device and a fluorescent imaging device are arranged under the same operation visual axis, and a space corresponding relation between the white light imaging device and the fluorescent imaging device is established, so that the white light imaging device and the fluorescent imaging device image the same operation field area; Setting a unified external trigger signal source, and simultaneously transmitting trigger signals to the white light imaging device and the fluorescent imaging device by the external trigger signal source in each acquisition period; When the trigger signal is received, the white light imaging device acquires a white light image at a corresponding moment; When the trigger signal is received, the fluorescence imaging device acquires a fluorescence signal by configuring an optical filter component matched with the wavelength of 1650nm and generates a fluorescence image; and giving the same time mark to the white light image and the fluorescent image acquired by the same trigger signal to obtain the white light image and the fluorescent image corresponding to the time mark.
  4. 4. The machine vision-based ovarian cancer intraoperative navigation method of claim 1, wherein the third step is specifically: converting the fluorescent image into a single-channel gray level image, and performing linear normalization processing on pixel values of the single-channel gray level image to obtain a normalized fluorescent image; performing discrete wavelet transform on the normalized fluorescence image, decomposing the normalized fluorescence image into a low frequency approximation subband and a plurality of high frequency detail subbands; For each high-frequency detail sub-band, calculating the absolute values of all wavelet coefficients in the high-frequency detail sub-band, and solving the median of the absolute values of the wavelet coefficients; calculating the absolute deviation between the absolute value of the wavelet coefficient and the median, and solving the median of the absolute deviation to be used as the denoising threshold value of the high-frequency detail sub-band; Setting zero to wavelet coefficient whose absolute value is smaller than the denoising threshold value in the high-frequency detail sub-band, and reserving wavelet coefficient whose absolute value is larger than or equal to the denoising threshold value; performing wavelet inverse transformation based on the processed wavelet coefficients, and reconstructing to obtain a fluorescence image after wavelet denoising; In the fluorescence image after wavelet denoising, summing all pixel values in each sliding window according to the sliding window with a set size, and dividing the sum by the total number of pixel points in the sliding window to obtain a local background intensity average value of the corresponding sliding window; Subtracting the local background intensity mean value corresponding to the sliding window where the pixel point is located from the pixel value of each pixel point in the fluorescence image after wavelet denoising to obtain a corrected pixel value; when the difference result is smaller than zero, setting the corresponding corrected pixel value to zero, and generating a fluorescence correction image based on all corrected pixel values; Extracting a brightness channel of the white light image, and marking a pixel point with a pixel value exceeding a set brightness threshold value in the brightness channel as a highlight area; Calculating pixel gradient amplitude values of the white light image, and marking pixel points with gradient amplitude values lower than a set gradient threshold value as low texture areas; carrying out pixel level combination on the highlight region and the low texture region to generate a non-reliable region pixel set; And assigning 0 to the pixels in the unreliable region pixel set, and assigning 1 to the rest pixels to generate a white light reliable region mask, wherein the pixels with the value of 0 represent the unreliable region, and the pixels with the value of 1 represent the reliable region.
  5. 5. The machine vision-based ovarian cancer intraoperative navigation method of claim 1, wherein the fourth step is specifically: Inputting the fluorescence correction image into an encoder, wherein the encoder comprises a multi-layer two-dimensional convolution unit and a downsampling unit, the two-dimensional convolution unit adopts a convolution kernel with the size of 3 multiplied by 3 and combines convolution operation with the step length of 1, and the downsampling unit adopts convolution operation with the step length of 2 to output characteristic diagrams of a plurality of resolution levels to form a current frame characteristic set; Inputting the current frame feature set into a hierarchical memory stack, wherein the hierarchical memory stack comprises a short-time memory layer and a long-term memory layer, and the short-time memory layer stores the current frame feature set with a set frame number in a first-in first-out mode; calculating a characteristic mean value of a current frame characteristic set corresponding to the same spatial position along a time dimension, and calculating element-by-element difference values between the characteristic of each time frame and the characteristic mean value; when the element-by-element difference value is smaller than a set change threshold value in a plurality of continuous time frames, writing the characteristic mean value of the corresponding space position into a long-term memory layer; Aligning the feature set stored in the long-term memory layer with the current frame feature set in space position, and combining to form a memory feature set; inputting the memory feature set into a bootstrap prompting ring, calculating the numerical variance of the corresponding feature in the continuous time frame for each spatial position in the memory feature set, setting the value of the corresponding spatial position to be 1 when the numerical variance is smaller than a set variance threshold, otherwise setting the value to be 0, and generating a stable position diagram; Calculating the sum of feature absolute values of the memory feature set at each spatial position along the channel dimension to obtain a response graph, performing element-by-element multiplication operation on the response graph and the stable position graph, and performing minimum-maximum normalization processing to obtain a prompt weight graph; Copying and expanding the prompt weight graph along the channel dimension into a weight set consistent with the channel number of the memory feature set, and performing element-by-element multiplication operation on the weight set and the memory feature set to obtain a prompt enhanced feature set; The method comprises the steps of inputting a prompt enhancement feature set into a two-domain representation decoder, wherein the two-domain representation decoder comprises a region decoding branch and a structural response decoding branch, the region decoding branch comprises a multi-layer deconvolution unit, the deconvolution unit adopts a convolution kernel with the size of 4 multiplied by 4 and an up-sampling operation with the step length of 2, and a region predicted value is output through the convolution kernel with the size of 1 multiplied by 1; assigning 1 to pixel points with the regional prediction value larger than or equal to the set prediction threshold value, otherwise, assigning 0 to obtain a tumor regional mask; the structure response decoding branch comprises a convolution unit with the size of 3 multiplied by 3 and a gradient enhancement unit, wherein the gradient enhancement unit respectively carries out differential operation on the prompt enhancement feature set in the horizontal direction and the vertical direction and outputs a boundary prediction value through a convolution kernel with the size of 1 multiplied by 1; splicing the regional predicted value and the boundary predicted value in the channel dimension to generate a joint predicted feature; and carrying out Softmax normalization calculation on the combined prediction features in the channel dimension to obtain a tumor confidence map.
  6. 6. The machine vision-based ovarian cancer intraoperative navigation method of claim 1, wherein the fifth step is specifically: performing gray level conversion on the white light image to obtain a white light gray level image; multiplying the white light gray level image by a white light reliable region mask pixel by pixel to obtain a white light effective region image; calculating pixel gradient amplitude values for the white light effective area image to generate a white light gradient response diagram; comparing the gradient amplitude of each pixel position with a set white light gradient threshold value based on the white light gradient response graph, setting the value of the corresponding pixel position in a white light structure chart to be the gradient amplitude when the pixel gradient amplitude is larger than or equal to the set white light gradient threshold value, setting the value of the corresponding pixel position in the white light structure chart to be 0 when the pixel gradient amplitude is smaller than the set white light gradient threshold value, and forming the white light structure chart by the value results of all the pixel positions; multiplying the fluorescence correction image with the tumor region mask pixel by pixel to obtain a fluorescence image masked by the tumor region mask; multiplying the masked fluorescence image of the tumor region by the tumor confidence map pixel by pixel, and weighting each pixel value according to the corresponding tumor confidence to obtain a confidence weighted fluorescence image; calculating pixel gradient amplitude values for the confidence weighted fluorescence image, and generating a fluorescence gradient response diagram; And comparing the gradient amplitude of each pixel position with a set fluorescence gradient threshold value based on the fluorescence gradient response diagram, setting the value of the corresponding pixel position in a fluorescence structural diagram as the gradient amplitude when the pixel gradient amplitude is larger than or equal to the set fluorescence gradient threshold value, setting the value of the corresponding pixel position in the fluorescence structural diagram as 0 when the pixel gradient amplitude is smaller than the set fluorescence gradient threshold value, and forming the fluorescence structural diagram by the value results of all the pixel positions.
  7. 7. The machine vision-based ovarian cancer intraoperative navigation method of claim 1, wherein the sixth step is specifically: In the white light structure diagram, a local neighborhood with the size of 5 multiplied by 5 is built by taking each non-zero pixel position as a center; Comparing the values of the pixel positions in the local neighborhood, and when the value of the central pixel position is larger than the values of the rest pixel positions in the local neighborhood, judging that the central pixel position meets the local maximum condition, and marking the central pixel position as a white light key point; traversing all pixel positions meeting local maximum conditions to obtain a white light key point set; and traversing the non-zero pixel positions in the fluorescence structural diagram by adopting a 5 multiplied by 5 local neighborhood which is the same as that of the white light structural diagram, and selecting all pixel positions meeting the local maximum condition as fluorescence key points to obtain a fluorescence key point set.
  8. 8. The machine vision-based ovarian cancer intraoperative navigation method of claim 1, wherein the seventh step is specifically: Taking each key point in the white light key point set and the fluorescent key point set as a center, respectively reading pixel values in a 5X 5 local adjacent area from the upper left corner to the lower right corner in the white light structure diagram and the fluorescent structure diagram according to the row priority order, and sequentially arranging the pixel values to form a one-dimensional numerical sequence as a feature vector of the corresponding key point; Inputting the feature vectors corresponding to the white light key point set and the feature vectors corresponding to the fluorescent key point set into a SuperGlue algorithm, respectively performing element-by-element multiplication operation on each feature vector in the white light key point set and each feature vector in the fluorescent key point set, and performing summation operation on multiplication results to obtain corresponding matching scores; Constructing a matching score matrix based on the matching scores between all the white light key points and the fluorescent key points, dividing each matching score in the matching score matrix by the matching score summation result of the row to obtain a row normalized matching score matrix, and dividing each matching score in the row normalized matching score matrix by the matching score summation result of the column to obtain a normalized matching score matrix; In the normalized matching score matrix, matching and screening are carried out by adopting a bidirectional consistency constraint rule of a SuperGlue algorithm, wherein the bidirectional consistency constraint rule is that for any key point in a white light key point set, if a corresponding fluorescent key point has the maximum normalized matching score in a corresponding row of the white light key point, and the white light key point also has the maximum normalized matching score in a corresponding column of the fluorescent key point, the white light key point and the fluorescent key point are judged to form a group of matching pairs; traversing all the key point corresponding relations meeting the bidirectional consistency constraint rule to obtain a cross-mode key point matching result; extracting pixel coordinates of each matching pair under a white light image coordinate system and a fluorescent image coordinate system based on the cross-modal key point matching result; Respectively calculating corresponding coordinate differences for all matched pixel coordinates, summing the coordinate differences, and averaging to obtain translation parameters between a white light image coordinate system and a fluorescent image coordinate system; Dividing the Euclidean distance of each matched pair under the white light image coordinate system by the Euclidean distance under the fluorescent image coordinate system to obtain a corresponding distance ratio; Carrying out summation operation on the distance ratio of all matched pairs, dividing the summation result by the number of the matched pairs to obtain scale parameters between a white light image coordinate system and a fluorescent image coordinate system; the scale parameters are acted on the pixel coordinates in the fluorescent image coordinate system, and the pixel coordinates in the fluorescent image are subjected to scale transformation; And adding the translation parameters to the pixel coordinates of the fluorescent image after the scale conversion, so that the pixel coordinates in the fluorescent image are aligned with a white light image coordinate system, and generating a registration mapping relation from the fluorescent image to the white light image.
  9. 9. The machine vision-based intra-operative navigation method for ovarian cancer of claim 1, wherein the step eight is specifically: based on the registration mapping relation from the fluorescent image to the white light image, extracting pixel coordinates under a fluorescent image coordinate system for each pixel position in the tumor region mask, and sequentially performing scale transformation and translation transformation according to the registration mapping relation to obtain corresponding mapping coordinates under the white light image coordinate system; Performing pixel-by-pixel addition operation on the mapped tumor region mask and the white light image at the corresponding coordinate positions to obtain a tumor region superposition graph; Extracting pixel coordinates under a fluorescent image coordinate system for each pixel position in the tumor confidence map, and performing scale transformation and translation transformation according to the registration mapping relation to obtain mapping coordinates under a corresponding white light image coordinate system; And carrying out pixel-by-pixel addition operation on the mapped tumor confidence map and the tumor region superposition map at the corresponding coordinate positions, and setting the pixel value as the maximum gray value when the addition result is greater than the maximum gray value, so as to generate the intra-operative navigation image.
  10. 10. The machine vision-based intra-ovarian cancer navigation system of claim 1, performing the machine vision-based intra-ovarian cancer navigation method of any one of claims 1 to 9, comprising the following modules: The dual-channel image acquisition module is used for synchronously acquiring a white light image and a fluorescent image through a white light imaging device and a fluorescent imaging device which are arranged under the same operation visual axis in the ovarian cancer operation process, and giving the same time mark to the white light image and the fluorescent image; the image preprocessing module is used for carrying out normalization processing and discrete wavelet transformation on the fluorescent image to obtain a fluorescent correction image, and carrying out reliable region identification on the white light image to obtain a white light reliable region mask; The tumor segmentation module is used for inputting the fluorescence correction image into an improved SAM2 model for tumor segmentation, wherein the improved SAM2 model comprises an encoder, a hierarchical memory stack, a bootstrap prompting ring and a two-domain representation decoder which are connected in sequence, and outputs a tumor region mask and a tumor confidence map; The structure construction module is used for generating a white light structure chart based on the white light image and the white light reliable region mask and generating a fluorescence structure chart based on the fluorescence correction image, the tumor region mask and the tumor confidence chart; The key point extraction module is used for extracting a white light key point set based on the white light structure diagram and extracting a fluorescent key point set based on the fluorescent structure diagram; the cross-modal matching and registering module is used for carrying out key point matching by utilizing a SuperGlue algorithm based on the white light key point set and the fluorescent key point set to generate a registering mapping relation from a fluorescent image to a white light image; And the navigation image generation module is used for mapping the tumor region mask and the tumor confidence map to a white light image coordinate system based on the registration mapping relation from the fluorescent image to the white light image to generate an intraoperative navigation image.

Description

Machine vision-based ovarian cancer intraoperative navigation method and system Technical Field The invention relates to the technical field of medical image processing, in particular to a machine vision-based ovarian cancer intraoperative navigation method and a machine vision-based ovarian cancer intraoperative navigation system. Background Along with the continuous increase of the requirements of accurate surgery and real-time navigation in operation, the high-precision positioning and boundary recognition technology for gynecological malignant tumors such as ovarian cancer is widely focused. The traditional ovarian cancer operation judgment mainly depends on subjective experience of medical staff on tissue color, morphology and touch sense under white light imaging, and auxiliary identification is carried out by assisting traditional near infrared fluorescence imaging under partial conditions, but the following problems generally exist in practical application: The method is characterized in that a white light image and a fluorescent image in operation are usually obtained by different imaging systems, visual axes are not completely consistent and a strict time synchronization mechanism is lacked, so that two types of images are deviated in space position and time dimension, accurate superposition of tumor region information in the white light image is affected, fluorescent signals are easily affected by tissue scattering, blood absorption and instrument shielding in a complex operation field environment, signal intensity fluctuation is obvious, signal to noise ratio is low, noise is difficult to effectively inhibit by a traditional simple filtering and background subtraction method while detail information is ensured, a tumor region segmentation result is unstable, aiming at the problem of spatial registration of a cross-modal image, the traditional method is mostly dependent on manual selection points or a matching algorithm based on gray similarity, mismatching is easily generated under the condition that the difference of white light and fluorescent imaging mechanisms is obvious, registration accuracy is insufficient, meanwhile, a traditional segmentation network is mostly based on reasoning of single-frame images, modeling capability of image time consistency in continuous operation is lacked, segmentation result jumping phenomena easily occur under visual angle change, tissue deformation and local reflection interference, and continuity and reliability of a navigation image are affected. Therefore, how to provide a machine vision-based ovarian cancer intraoperative navigation method and system is a problem to be solved by those skilled in the art. Disclosure of Invention The invention aims to provide an ovarian cancer intraoperative navigation method and system based on machine vision, and the method combines an improved SAM2 model and a SuperGlue algorithm to construct a multi-mode image collaborative processing flow, so that accurate segmentation and cross-mode space registration of a tumor area in the ovarian cancer surgery are realized, stable mapping of fluorescence information to a white light visual field is completed, and the method has the advantages of high segmentation stability, high registration precision and visual and reliable navigation display, and can effectively improve the identification accuracy and operation safety in the surgery. According to the embodiment of the invention, the ovarian cancer intraoperative navigation method based on machine vision comprises the following steps of: injecting a tumor microenvironment response type NIR-IIb composite nano probe into a subject to be operated before ovarian cancer operation to obtain a fluorescence signal; synchronously acquiring a white light image and a fluorescence image corresponding to the fluorescence signal at the same time point in the ovarian cancer operation process to obtain a white light image and a fluorescence image; step three, carrying out normalization processing and discrete wavelet transformation on the fluorescent image to obtain a fluorescent correction image, and carrying out reliable region identification on the white light image to obtain a white light reliable region mask; Inputting the fluorescence correction image into an improved SAM2 model for tumor segmentation, wherein the improved SAM2 model comprises an encoder, a hierarchical memory stack, a bootstrap prompting ring and a two-domain representation decoder which are connected in sequence, and outputting a tumor region mask and a tumor confidence map; Generating a white light structure chart based on the white light image and the white light reliable region mask, and generating a fluorescence structure chart based on the fluorescence correction image, the tumor region mask and the tumor confidence chart; Step six, extracting a white light key point set based on the white light structure diagram, and extracting a fluorescent key point set based on