CN-121622100-B - Multi-mode data phase synchronization method for heart sounds, electrocardiograms and ultrasonic cardiography

CN121622100BCN 121622100 BCN121622100 BCN 121622100BCN-121622100-B

Abstract

The invention discloses a multi-mode data phase synchronization method for heart sounds, electrocardiograms and ultrasonic cardiography, which belongs to the technical field of medical image processing and comprises the following steps of A1, obtaining original ultrasonic cardiography video and independently recorded heart sound audio data information, A2, preprocessing the obtained data and automatically constructing samples, A3, constructing a dual-path neural network architecture, extracting electrocardiosignal shapes, obtaining a digital electrocardiosignal sequence, A4, identifying periodic characteristic points of heart sounds and electrocardiosignals based on the digital electrocardiosignals, and A5, aligning asynchronous data based on linear phase resampling. According to the invention, the video frames, the electric signals and the heart sound signals under different sampling rates are forcedly mapped into a unified phase coordinate system through a linear interpolation algorithm, so that absolute synchronization of the three on the physiological logic of heart beat is finally realized, and the influence of individual heart rate difference on data alignment is eliminated.

Inventors

BAI YAN
WANG XIANMIN
TANG YUYING
MA LIQIONG
YUAN TAO
XIONG WEIPING
ZHAO JING
LUO LIQIONG
ZHANG YAHUI

Assignees

四川省妇幼保健院

Dates

Publication Date: 20260505
Application Date: 20260205

Claims (6)

1. The method for synchronizing the phases of the multi-mode data of the heart sounds, the electrocardiograms and the ultrasonic cardiography is characterized by comprising the following steps of: A1. acquiring original ultrasonic heart image video and independently recorded heart sound audio data information; A2. Preprocessing the acquired data and carrying out sample automatic construction; A3. Constructing a dual-path neural network architecture, extracting electrocardiographic waveforms, and obtaining a digital electrocardiographic signal sequence; A4. Based on the digital electrocardiosignal, identifying the periodic characteristic points of heart sounds and electrocardiosignals; A5. aligning the asynchronous data based on linear phase resampling; the step A2 specifically comprises the following substeps: A21. Analyzing the acquired image and video stream data, and generating a binary mask through HSV color filtering; A22. the sliding window bidirectional tracking waveform track which introduces motion trend prediction is adopted, namely, a first waveform pixel point which meets a color threshold value is positioned in a binary mask through scanning column by column, is used as an initial seed point, forward tracking is carried out along the increasing direction of the horizontal coordinate of an image by taking the initial seed point as the center until the image edge is reached, and then the initial seed point is returned to carry out backward tracking along the decreasing direction of the horizontal coordinate, so that the whole waveform track is ensured to be completely obtained; A23. Outputting an initial numerical sequence with a breakpoint mark, and generating a negative sample training set from samples which are failed to be extracted; When the effective pixel density detected by the sliding window in the detection area is lower than a preset continuity threshold value, indicating that character shielding or pixel breakage exists at the position, automatically marking the frame sequence number as a breakpoint to be repaired, and outputting an initial value sequence with defect characteristics; the construction of the dual-path neural network architecture in the step A3 specifically comprises the following substeps: A31. a confidence judging module is arranged to judge the effectiveness of the image area; A32. Capturing the dependence of the waveform position at the predicted moment on the history and future tracks based on the autocorrelation of the electrocardiosignal; A33. The dependency relationship of the history and future tracks before and after capturing the prediction in the step A32 is aggregated, and the reconstruction is carried out in a pixel fracture area; A34. Outputting a digital electrocardiosignal sequence which is continuous on a time axis and aligned with the video frame rate; the step A4 specifically comprises the following substeps: A41. The method comprises the steps of highlighting waveform slopes of a digital electrocardiosignal sequence by using a five-point derivative filter, and identifying physiological anchor points; A42. Filtering out environmental noise through band-pass filtering of 20-500Hz, and extracting a heart sound anchor point; A43. Performing time axis anchoring based on a valve closing frame in a physiological anchor point, a heart sound anchor point and an ultrasonic heart sound image video; A44. Outputting a synchronization matrix containing the aligned image, the electrocardio and heart sound data; The step A5 specifically comprises the steps of defining each detected adjacent R-R interval or adjacent S1-S1 interval as a standardized cardiac cycle interval, and carrying out phase mapping to match and align an ultrasonic electrocardiographic image video, an electrocardiographic signal and a heart sound signal with a sampling rate, wherein R represents an identified electrocardiographic anchor point, and S1 represents an extracted heart sound anchor point.
2. The method for phase synchronizing heart sounds, electrocardiograms and echocardiography multi-mode data according to claim 1, wherein the calculation formula for generating the binarization mask by HSV color filtering in the step a21 specifically comprises: ; Wherein Mask (x, y) represents a binary Mask output at coordinates (x, y) for determining whether the point belongs to a target pixel, wherein 1 is the target pixel and 0 is the background pixel, H, S, V represents component values of the pixel point of the input image at coordinates (x, y) in hue, saturation and brightness channels respectively, and [ H min ,H max ]、[S min ,S max ]、[V min ,V max ] represents preset color threshold intervals on the corresponding hue, saturation and brightness channels respectively for locking waveform colors and filtering background noise.
3. The method for phase synchronizing heart sounds, electrocardiographs and echocardiography multi-modal data according to claim 1, wherein the determining function for determining the validity of the image region in the step a31 is: ; Where w i denotes the i-th partial image sliding window of the input, F conv denotes the feature extraction convolution operation, θ c denotes the network learning parameter, σ denotes the activation function mapping the output to the [0,1] interval, and C (w i ) denotes the confidence score of the i-th partial image sliding window; The step A31 is that the effectiveness of the image area is judged specifically, when the confidence score is more than or equal to 0.8, the current image area track is considered to be effective.
4. The method for phase synchronizing heart sounds, electrocardiograms and ultrasonic cardiography multi-mode data according to claim 3, wherein the calculation formula of the dependence of the time waveform position on the history and future tracks in the step A32 is as follows: ; Wherein Y t represents the waveform ordinate amplitude to be predicted at the time t, X t represents the characteristic input at the current time, H t-1 and H t+1 represent the hidden layer states at the history and future times respectively, Representing a nonlinear mapping function, θ t representing a network learning parameter; In the step A33, the dependency relationship of the history and the future track is multiplied point by point at the time before and after capturing and predicting through a gating fusion formula, and the repaired track point is generated and reconstructed in a pixel fracture area; The gating fusion formula specifically comprises output=w×v, wherein Output represents the repaired track point, w represents the waveform predicted value, and v represents the validity mask.
5. The method for phase synchronizing heart sounds, electrocardiograms and echocardiography multi-modal data according to claim 1, wherein the calculation formula of the five-point derivative filter in the step a41 is as follows: ; wherein x n represents the input digital sampling point, y n represents the differential signal after five-point derivative filtering; The step A42 specifically extracts heart sound anchor points through average shannon energy, and the calculation formula is as follows: ; where E shannon represents the calculated energy envelope value, N represents the total number of samples in the calculation window, and x i represents the amplitude value of the i-th sample in the window.
6. The method for phase synchronizing heart sounds, electrocardiograms and echocardiography multi-modal data according to claim 1, wherein the calculation formula of the phase map is: ; Wherein phi (T) represents the phase percentage of the time T in the standardized cardiac cycle, T represents the time stamp of the current sampling point, T S represents the starting anchor point of the current cardiac cycle, namely the peak time of the current cardiac anchor point or the starting time of the cardiac sound anchor point, T e represents the starting anchor point of the next cardiac cycle, and T e -T S represents the absolute duration of the cardiac cycle.

Description

Multi-mode data phase synchronization method for heart sounds, electrocardiograms and ultrasonic cardiography Technical Field The invention relates to the technical field of medical image processing, in particular to a method for synchronizing the phases of heart sounds, electrocardiograms and ultrasonic cardiography multi-mode data. Background Congenital heart disease (Congenital HEART DISEASES, CHD) is a congenital deformity caused by abnormal development of heart and large blood vessels in embryo phase, and early discovery and early treatment are critical for reducing the risk of serious complications such as heart failure, cardiogenic shock and the like. At present, clinical diagnosis of the congenital heart disease mainly depends on physical examination, echocardiography, electrocardiogram, heart sound auscultation and other means. With the development of deep learning technology, automatic disease classification recognition based on echocardiography videos has become a research hotspot. In order to enhance the influence of the heart periodicity and improve the recognition accuracy, it is important to perform multi-modal alignment and information complementation by fusing Electrocardiogram (ECG) signals. In the prior art, it is generally necessary to extract the built-in electrocardiographic waveforms from the keyframes of the echocardiographic video. However, the existing extraction methods mainly have the following problems and disadvantages: 1. the embedded waveform extraction is highly dependent on the color threshold-traditional algorithms separate the electrocardiographic pixels primarily by setting a threshold for a particular color (e.g., green) in the ultrasound video frame. This approach is extremely sensitive to color deviations and is difficult to accommodate for waveform color variations in different devices and acquisition environments, resulting in less robust extraction. 2. The pixel level extraction has poor anti-interference capability and is easy to generate position offset, namely the existing algorithm is easy to be interfered when processing characters, scales or echo noise in an ultrasonic image, so that the positioning deviation of the vertical axis coordinates (amplitude positions) of the electrocardiographic waveform is easy to be caused. Such pixel-level estimation algorithms lack noise immunity and often result in distortion of the extracted electrocardiographic signals. 3. The lack of time sequence consistency modeling is that the conventional mathematical modeling algorithm often ignores the continuity and logic dependence of the electrocardiographic waveform along with the time change, and the lack of effective utilization of time sequence information. 4. It is difficult to deal with waveform breakage and occlusion problems-the prior art is based primarily on the assumption that the pixel domain is continuous. When the electrocardiographic waveform in the echocardiogram is interrupted due to discontinuous pixels, occlusion by other text marks or color reduction, the traditional algorithm (such as sliding window search) can cause extraction failure due to reaching a termination condition, and the problem of waveform missing or incomplete is generated. 5. Asynchronous data synchronization is difficult, namely when heart sound signals and echocardiographic data are acquired non-simultaneously (asynchronously), due to the lack of an effective electrocardio reference extraction means and deep excavation of heart beat periodic ripple characteristics, accurate alignment of multi-source heterogeneous data on heart beat periodic phases is difficult to realize. The existence of the defects leads to rejection of a large amount of ultrasonic data with waveform defects, causes waste of data resources, and seriously affects performance evaluation and clinical practicality of the multi-mode auxiliary diagnostic model. Therefore, how to accurately and completely extract the electrocardiographic data from the complex ultrasonic video and realize the reliable alignment of the electrocardiographic data and other modal data such as heart sounds is a technical problem to be solved in the current field. Disclosure of Invention In order to solve the above problems, the present invention provides a method for phase synchronizing multi-modal data of heart sounds, electrocardiograms and echocardiography, comprising the following steps: A1. acquiring original ultrasonic heart image video and independently recorded heart sound audio data information; A2. Preprocessing the acquired data and carrying out sample automatic construction; A3. Constructing a dual-path neural network architecture, extracting electrocardiographic waveforms, and obtaining a digital electrocardiographic signal sequence; A4. Based on the digital electrocardiosignal, identifying the periodic characteristic points of heart sounds and electrocardiosignals; A5. the asynchronous data is aligned based on linear phase resampling. Furt