CN-122000011-A - Abdomen operation video semiautomatic labeling method based on mark point tracking retrospective anatomic path

CN122000011ACN 122000011 ACN122000011 ACN 122000011ACN-122000011-A

Abstract

The invention provides a semi-automatic labeling method for abdominal operation video based on a marked point tracking retrospective anatomic path, and belongs to the technical field of operation video labeling. The method comprises the steps of S1, constructing a path internal data set, S2, constructing a path external data set, S3, constructing an environment interference coefficient of an ith mark point, constructing a path reliability coefficient of the ith mark point, S4, constructing a deep learning model for video annotation, constructing a path comprehensive correction coefficient of the ith mark point, S5, dividing a surgical video into a plurality of anatomical region video segments, processing the video segments of each anatomical region according to video frame time in an inverted sequence, performing backtracking correction on the position of the mark point according to the path comprehensive correction coefficient of the ith mark point, and combining a memory module to realize cross-frame matching and updating to generate a continuous anatomical path backtracking annotation result. The invention effectively replaces the manual complete marking mode and improves the marking accuracy.

Inventors

LI JIAXIN
XIAO WEIDONG
WANG ZIHAN
JIANG ENLAI
ZHENG DAOFENG
QIN JIAJIA

Assignees

中国人民解放军陆军军医大学第二附属医院

Dates

Publication Date: 20260508
Application Date: 20260121

Claims (10)

1. The abdomen operation video semiautomatic labeling method based on the marked point tracking retrospective anatomic path is characterized by comprising the following steps: S1, acquiring a laparoscopic abdominal operation video, setting a plurality of mark points at key positions on two sides of a target anatomical path by an expert in an operation stage of the confirmed target anatomical path, acquiring spatial coordinate information of an ith mark point in a current video frame, spatial distance between adjacent mark points and local texture characteristic parameters of tissues around the ith mark point, constructing an initial feature vector of the mark point based on the spatial coordinate information and the local texture characteristic parameters, and constructing a path internal data set; S2, after the path internal data set is constructed, continuously acquiring visual environment parameters which influence the recognition stability of the anatomical path, wherein the visual environment parameters comprise the proportion of pixels, the range of variation of a lens visual angle, the fluctuation value of local illumination intensity and the interference frequency of the instrument, which are shielded by the instrument, of tissues in a target area, and constructing the path external data set; S3, constructing an environmental interference coefficient of the ith mark point based on the path internal data set Constructing a path credibility coefficient of the ith mark point based on the path external data set ; S4, constructing a deep learning model for video annotation by utilizing the initial feature vector of the ith mark point and the convolutional neural network, and combining the environmental interference coefficient of the ith mark point And the path confidence coefficient of the ith mark point Constructing a path comprehensive correction coefficient of the ith mark point ; S5, dividing the operation video into a plurality of anatomic region video segments, processing each anatomic region video segment according to the video frame time reverse order, and synthesizing correction coefficients according to the path of the ith mark point And backtracking correction is carried out on the positions of the marking points, cross-frame matching and updating are realized by combining the memory module, and continuous anatomic path backtracking marking results are generated.
2. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths by using labeled points according to claim 1, wherein the step S1 comprises: S11, performing multi-angle real-time shooting on laparoscopic surgery by placing a plurality of laparoscopic cameras, and obtaining laparoscopic abdominal surgery videos; S12, based on the laparoscopic abdominal operation video, selecting a key video frame containing a target anatomical path from the laparoscopic abdominal operation video as a marking frame, converting the video into an image, manually clicking key positions on two sides of the target anatomical path by using a marking tool on an image interface of the marking frame by an expert to generate a plurality of marking points, and recording two-dimensional image coordinate information of the ith marking point in the current video frame through a two-dimensional coordinate system, wherein the two-dimensional image coordinate information comprises transverse pixel coordinates of the ith marking point under the image coordinate system And vertical pixel coordinates ; S13, numbering the mark points according to the trend sequence of the anatomical path based on the two-dimensional image coordinate information of the ith mark point in the current video frame, defining two mark points with adjacent numbers as a group of adjacent mark point pairs, calculating the geometric distance between the ith mark point and the (i+1) th mark point by adopting a Euclidean distance formula, and obtaining the space distance parameter between the ith mark point and the (i+1) th mark point ; S14, taking the coordinate position of the ith marking point as the center, cutting a local image area with a fixed size in a corresponding video frame as a neighborhood image block of the marking point, carrying out noise suppression on median filtering of the neighborhood image block, converting the neighborhood image block from an RGB color space to an HSV color space, and then extracting local texture feature parameters reflecting the tissue texture distribution, the texture direction and the local structural feature of the ith marking point based on the processed neighborhood image block so as to enhance the characterization capability of the tissue texture difference; S15, based on the two-dimensional image coordinate information of the ith mark point in the current video frame and the local texture feature parameters of the tissue texture distribution, texture direction and local structural feature of the ith mark point, carrying out standardization processing on the two-dimensional image coordinate information of the ith mark point in the current video frame, carrying out standardization processing on the tissue texture distribution, texture direction and local texture feature parameters of the local structural feature of the ith mark point, carrying out fusion on the two-dimensional image coordinate information of the treated ith mark point in the current video frame and the tissue texture distribution, texture direction and local texture feature parameters of the ith mark point, and constructing an initial feature vector of the ith mark point An intra-path dataset is constructed.
3. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths by using labeled points according to claim 2, wherein the step S2 comprises: s21, dividing a surgical instrument area and a tissue area in a video image based on a laparoscopic abdominal operation video, distinguishing the position distribution situation of the instrument and the anatomical tissue, and then counting the proportion of pixels in the target area, in which the tissue is blocked by the instrument, by taking the area in which the ith mark point is positioned as the target area to obtain the blocking interference intensity of the ith mark point ; S22, in continuous video frames of a laparoscopic abdominal operation video, analyzing the overall picture change condition between adjacent video frames, detecting and shielding an instrument area in the video frames, extracting stable image feature points in a non-instrument area in the video and performing inter-frame matching, establishing a global image transformation relation between the adjacent video frames, extracting view angle change features reflecting lens translation change, rotation change and view field scaling change, performing scale unification processing on the view angle change features, and then performing maximum change value processing on the view angle change features according to a preset fusion rule to obtain a lens view angle change amplitude parameter for representing the view angle change intensity of a lens, thereby obtaining view angle change interference intensity V of the current video frame; S23, taking a local area where the ith mark point is positioned as an illumination analysis area, intercepting a neighborhood image block containing the ith mark point in a corresponding video frame, converting the neighborhood image block into a color space with more stable brightness components to extract brightness information reflecting illumination intensity, and then carrying out statistical analysis on brightness distribution conditions in the neighborhood image block of the ith mark point to obtain brightness fluctuation values of the local area where the ith mark point is positioned in the current video frame to obtain illumination fluctuation intensity of the ith mark point ; S24, continuously detecting instruments in the surgical video to obtain the spatial position distribution condition of the instruments in each video frame, taking a neighborhood region where an ith mark point is positioned as an interference judging region, and marking the current moment as an instrument interference event when the instruments are detected to enter the current neighborhood region to block tissues to obtain the instrument interference intensity of the ith mark point And constructing a path external data set.
4. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths by using labeled points according to claim 3, wherein the step S3 comprises: S31, obtaining an environment interference coefficient based on the path internal data set in the following way; first, based on the lateral pixel coordinates of the ith mark point in the image coordinate system And vertical pixel coordinates Obtaining the disturbance component of coordinates by calculation ; Subsequently using the parameter of the spatial distance between the i-th mark point and the i+1-th mark point Obtaining geometrical disturbance components by calculation ; Next, the initial feature vector of the ith marker point is used Calculating the variation of the feature vector between the ith mark point and the (i+1) th mark point ; Disturbance component to the coordinate Component of geometric disturbance And variation of feature vector between the i-th mark point and the i+1-th mark point Normalization processing is carried out to obtain a coordinate disturbance component after normalization processing Component of geometric disturbance And variation of feature vector between the i-th mark point and the i+1-th mark point And the environmental interference coefficient of the ith mark point is obtained by calculation according to the following formula ; ; In the formula, 、 And Respectively, weight coefficients.
5. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths with labeled points according to claim 4, wherein the step S3 further comprises: s32, presetting an environment interference threshold Q, and setting the environment interference coefficient of the ith mark point In contrast to the ambient interference threshold Q, comprising: When (when) When > Q, it indicates that the visual environment of the ith mark point in the current video frame is abnormal, an enhanced backtracking correction policy needs to be executed on the ith mark point, including: When Q < When the Q is less than or equal to 1.2, 20% -40% of the position correction is carried out on the position of the ith mark point; when 1.2 is Q < When the Q is less than or equal to 1.5, 41-70% of the position correction is carried out on the position of the ith mark point; When (when) At >1.5 x Q, 71% -90% position correction is performed on the i-th mark point position When (when) And when Q is less than or equal to Q, the visual environment of the ith mark point in the current video frame is normal.
6. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths with labeled points according to claim 5, wherein the step S3 further comprises: s33, obtaining a path credibility coefficient of an ith mark point based on the path external data set in the following way; first, the shielding interference intensity of the ith mark point View angle variation interference intensity V of current video frame and illumination fluctuation intensity of ith mark point Instrument interference intensity for ith mark point Carrying out normalization processing to obtain the shielding interference intensity of the ith mark point after normalization processing View angle variation interference intensity for current video frame Intensity of illumination fluctuation of i-th mark point Instrument interference intensity for ith mark point ; Second, based on the shielding interference intensity of the ith mark point View angle variation interference intensity for current video frame Intensity of illumination fluctuation of i-th mark point Instrument interference intensity for ith mark point Calculating to obtain the comprehensive interference quantity of the ith mark point ; Finally, the comprehensive interference quantity of the ith mark point is utilized Calculating to obtain the path credibility coefficient of the ith mark point ; 。
7. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths with labeled points according to claim 6, wherein the step S3 further comprises: S34, through a preset path credibility threshold R, the path credibility coefficient of the ith mark point is obtained In contrast to the path reliability threshold R, comprising: When (when) When R is greater than the threshold value, the path positioning reliability of the ith mark point in the current video frame is normal; When (when) And when R is less than or equal to R, indicating that the path positioning reliability of the ith mark point in the current video frame is abnormal, and performing a slight backtracking correction operation on the ith mark point, wherein the method comprises the following steps of: When R < > When the mark point position is less than or equal to 1.2 x R, 5% -15% of position adjustment is carried out on the i-th mark point position; When (when) And >1.2 x r, the i-th mark point position is kept unchanged.
8. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths with labeled points according to claim 7, wherein the step S4 comprises: s41, constructing a deep learning model for video annotation by using an i-th marker point initial feature vector and a convolutional neural network, training and testing the deep learning model by using a path internal data set and a path external data set, taking the trained deep learning model as an operation video marker evaluation model, simultaneously taking middle layer output of equipment operation deep learning model as a feature vector to identify feature information, and taking the trained deep learning model as data operation prediction.
9. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths with labeled points according to claim 8, wherein the step S4 further comprises: S42, the environmental interference coefficient of the ith mark point Path confidence coefficient with the i-th marker point In association, the path integrated correction coefficient of the ith mark point is obtained by calculation according to the following formula ; 。
10. The method for video semiautomatic labeling of abdominal surgery based on tracking backward anatomical paths by using labeled points according to claim 9, wherein the step S5 comprises: S51, dividing a laparoscopic abdominal operation video into a plurality of anatomic region video segments, processing the video segments in each anatomic region video segment frame by frame in a reverse order according to time by taking a last frame as a starting frame, performing mark point cross-frame matching on each precursor video frame based on a memory module, and evaluating the path comprehensive correction coefficient of an ith mark point Determining the correction requirement, performing retrospective correction on the positions of the mark points, and updating the memory module until retrospective marking of all video frames of the current segment is completed; s52, through presetting a path comprehensive correction threshold Z, and integrating the path correction coefficient of the ith mark point Comparing with a path comprehensive correction threshold Z, comprising: When (when) When > Z, it indicates that the path of the ith mark point in the current video frame is abnormal, and backtracking correction operation needs to be performed on the ith mark point, including: When Z < > When the Z is less than or equal to 1.2, 10% -30% of the position correction is carried out on the position of the ith mark point; When 1.2 x Z < When the Z is less than or equal to 1.5, the position of the ith mark point is corrected by 32 to 60 percent, when When 1.5 x Z, performing 62% -90% position correction on the ith mark point; When (when) And when Z is less than or equal to the value, the path positioning of the ith mark point is normal, and the current position of the ith mark point is kept unchanged.

Description

Abdomen operation video semiautomatic labeling method based on mark point tracking retrospective anatomic path Technical Field The invention relates to the technical field of surgical video marking, in particular to an abdominal surgical video semiautomatic marking method based on marking point tracking backtracking anatomical paths. Background The laparoscopic abdominal operation has the advantages of small trauma, quick recovery and the like, and is widely applied to various clinical operation scenes such as rectal cancer, colorectal cancer, gastric cancer, liver and gall diseases and the like, and in the operation process, doctors generally need to separate and operate along a predetermined anatomical path so as to avoid injuring important blood vessels, nerves and key tissue structures by mistake, so that the anatomical path is accurately marked in an operation video, and the laparoscopic abdominal operation has important significance for intra-operation navigation, postoperative multiple disc analysis, operation training and intelligent auxiliary operation system construction. The existing anatomical path labeling mode mainly relies on manual operation video frame by frame for labeling, the labeling process is long in time consumption and high in workload, the professional experience requirements on labeling personnel are high, large-scale data processing and clinical application requirements are difficult to meet, textures and colors among different tissues are highly similar under the field of view of a laparoscope, and an anatomical path is not a single organ boundary but an operation route formed by relying on experience of a doctor, so that an anatomical path structure is difficult to accurately describe, and the abdominal operation video semiautomatic labeling method based on the labeled point tracking retrospective anatomical path is provided. Disclosure of Invention In order to overcome the defects, the invention provides a semi-automatic labeling method for abdominal operation video based on the marked point tracking retrospective anatomical path, which overcomes the technical problems or at least partially solves the problems. The invention is realized in the following way: The invention provides a method for semi-automatically labeling abdominal operation video based on a marked point tracking retrospective anatomic path, which comprises the following steps: S1, acquiring a laparoscopic abdominal operation video, setting a plurality of mark points at key positions on two sides of a target anatomical path by an expert in an operation stage of the confirmed target anatomical path, acquiring spatial coordinate information of an ith mark point in a current video frame, spatial distance between adjacent mark points and local texture characteristic parameters of tissues around the ith mark point, constructing an initial feature vector of the mark point based on the spatial coordinate information and the local texture characteristic parameters, and constructing a path internal data set; S2, after the path internal data set is constructed, continuously acquiring visual environment parameters which influence the recognition stability of the anatomical path, wherein the visual environment parameters comprise the proportion of pixels, the range of variation of a lens visual angle, the fluctuation value of local illumination intensity and the interference frequency of the instrument, which are shielded by the instrument, of tissues in a target area, and constructing the path external data set; S3, constructing an environmental interference coefficient of the ith mark point based on the path internal data set Constructing a path credibility coefficient of the ith mark point based on the path external data set; S4, constructing a deep learning model for video annotation by utilizing the initial feature vector of the ith mark point and the convolutional neural network, and combining the environmental interference coefficient of the ith mark pointAnd the path confidence coefficient of the ith mark pointConstructing a path comprehensive correction coefficient of the ith mark point; S5, dividing the operation video into a plurality of anatomic region video segments, processing each anatomic region video segment according to the video frame time reverse order, and synthesizing correction coefficients according to the path of the ith mark pointAnd backtracking correction is carried out on the positions of the marking points, cross-frame matching and updating are realized by combining the memory module, and continuous anatomic path backtracking marking results are generated. In a preferred embodiment, the step S1 includes: S11, performing multi-angle real-time shooting on laparoscopic surgery by placing a plurality of laparoscopic cameras, and obtaining laparoscopic abdominal surgery videos; S12, based on the laparoscopic abdominal operation video, selecting a key video frame containing a target anatomical path