Key Point-Guided Temporal Network Used for Segmentation of Echocardiography
Xiang Zhuo1, Chen Weiling2, Tian Xiaoyu1, Zhao Cheng1, Wang Tianfu1, Lei Baiying1*
1(Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging,School of Biomedical Engineering, Shenzhen University MedicalSchool, Shenzhen University, Shenzhen 518060, Guangdong, China) 2(Department of Ultrasound Department, Shenzhen Children Hospital, Hospital of Shantou University, Shenzhen 518050, Guangdong, China
Abstract：Echocardiographic segmentation is an important step in the screening of congenital heart disease. However, the quality of echocardiogram image usually is relatively low, and some key heart structures in the echocardiogram video portion of the frame can blur or disappear due to the beating of the heart. For the target frame whose structure disappears, it is usually necessary to deduce the position of the key structure in the target frame by relying on other frames with clear structure in the echocardiography video. Aiming to address these challenges, this study designed a key point guided timing network to complete the segmentation of echocardiography. Specifically, for the target frame to be segmented, other frames in the ultrasonic video were used as secondary frames. First, a bidirectional temporal network (BTN) was designed to extract the structure information from the auxiliary frame, and in this process, the key points guided the network to extract the key structure information. Then, a transformer temporal attention (TTA) module was proposed to adjust the feature weights of each auxiliary frame and focus on the auxiliary frame with clear structure. In addition, this study proposed an image mapping (IM) module, which mapped the structure information of the auxiliary frame directly to the target frame and completed the supplement of missing structure information in the target frame. In this study, experiments were conducted on the parasternal short axis section data of 98 cases, and the average Dice reached 0.8269. Experimental results showed that the proposed method could be effectively applied to echocardiogram segmentation.
[1] Zimmerman MS, Smith AGC, Sable CA, et al. Global, regional, and national burden of congenital heart disease, 1990–2017: a systematic Analysis for the Global Burden of Disease Study 2017[J]. The Lancet Child & Adolescent Health, 2020, 4:185-200. [2] Avendi MR, Kheradvar A, Jafarkhani H. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac mri[J]. Medical Image Analysis, 2016, 30:108-119. [3] Ngo TA, Lu Z, Carneiro G Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance[J]. Medical Image Analysis, 2017, 35:159-171. [4] Guo Libao, Lei Baiying, Chen Weiling, et al. Dual attention enhancement feature fusion network for segmentation and quantitative analysis of paediatric echocardiography[J]. Medical Image Analysis, 2021, 71:102042. [5] Li Kai, Wang Shujun, Yu Lequan, et al. Dual-teacher++: exploiting intra-domain and inter-domain knowledge with reliable transfer for cardiac segmentation[J]. IEEE Transactions on Medical Imaging, 2020, 40(10): 2771-2782. [6] Zhao Cheng, Chen Weiling, Qin Jin, et al. IFT-net: interactive fusion transformer network for quantitative analysis of pediatric echocardiography[J]. Medical Image Analysis, 2022, 82:102648. [7] Leclerc S, Smistad E, Østvik A, et al. Lu-net: a multistage attention network to improve the robustness of segmentation of left ventricular structures in 2-D echocardiography[J]. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 2020, 67:2519-2530. [8] Hu Yujin, Xia Bei, Mao Muyi, et al. Aidan: an attention-guided dual-path network for pediatric echocardiography segmentation[J]. IEEE Access, 2020, 8:29176-29187. [9] Yu Changqian, Wang Jjingbo, Chao Peng, et al. Bisenet: bilateral segmentation network for real-time semantic segmentation[C] // Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 325-341. [10] Valanarasu JMJ, Oza P, Hacihaliloglu I, et al. Medical transformer: gated axial-attention for medical image segmentation[C] // Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference. Strasbourg: MICCAI, 2021: 36-46. [11] 吴宣言, 缑新科, 朱子重,等. 深层聚合残差密集网络的超声图像左心室分割[J]. 中国图象图形学报, 2020, 25(9):1930-1942. [12] 葛帅, 严加勇, 谢利剑,等. 改进型 U-Net 网络的左心室超声心动图像分割[J]. 软件导刊, 2021, 20(2): 206-209. [13] 胡玉进, 雷柏英, 郭力宝,等. 基于 bisenet 的小儿超声心动图左心分割方法[J]. 中国生物医学工程学报, 2019, 38(5): 533-539. [14] 俊庞, 永雄王, 丽君陈,等. 面向儿科超声心动图双侧心室分割的注意力引导网络[J]. 生物医学工程学杂志, 2023, 40(5):928-937. [15] Ke Sun, Xiao Bin, Liu Dong, et al. Deep high-resolution representation learning for human pose estimation[C] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: CVPR, 2019: 5686-5696. [16] Cao Zhe, Simon T, Wei SE, et al. Realtime multi-person 2d pose estimation using part affinity fields[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: CVPR, 2017: 7291-7299. [17] Payer C, Štern D, Bischof H, et al. Regressing heatmaps for multiple landmark localization using cnns[C]// Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference. Athens: MICCAI, 2016: 230-238. [18] Ta KMT. Multi-task learning for cardiac motion analysis and segmentation in echocardiography [D]. New Haven: Yale University, 2023. [19] Dozen A, Komatsu M, Sakai A, et al. Image segmentation of the ventricular septum in fetal cardiac ultrasound videos based on deep learning using time-series information[J]. Biomolecules, 2020, 10(11): 1526. [20] Chen Zejian, Wei Zhuo, Wang Tianfu, et al. Semi-supervised representation learning for segmentation on medical volumes and sequences[J]. IEEE Transactions on Medical Imaging, 42(12): 3972-3986, 2023. [21] Cho K, Van Merriënboer B, Gulcehre C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/OL]. https://arxiv.org/abs/1406.1078, 2014-09-03/2024-01-03. [22] Wang Xinyao, Bo Liefeng, Li Fuxin. Adaptive wing loss for robust face alignment via heatmap regression[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: ICCV, 2019: 6971-6981. [23] Hu Jie, Shen Li, Sun Gang. Squeeze-and-excitation networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: CVPR, 2018: 7132-7141. [24] Wang Qilong, Wu Banggu, Zhu Pengfei, et al. Eca-net: efficient channel attention for deep convolutional neural networks[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: CVPR, 2020: 11534-11542. [25] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 6000-6010. [26] Wang Panqu, Chen Pengfei, Yuan Ye, et al. Understanding convolution for semantic segmentation[C]// 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe: WACV, 2018: 1451-1460. [27] Chen LC, Zhu Yukun, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]// Proceedings of the European Conference on Computer Vision (ECCV). Munich: ECCV, 2018: 801-818. [28] Zhao Hengshuang, Shi Jianping, Qi Xiaojuan, et al. Pyramid scene parsing network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: CVPR, 2017: 2881-2890. [29] Fu Jun, Liu Jing, Tian Haijie, et al. Dual attention network for scene segmentation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: CVPR, 2019: 3146-3154. [30] Badrinarayanan V, Kendall A, Cipolla R Segnet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39:2481-2495. [31] Dong Shunjie, Zhao Jinlong, Zhang Maojun, et al. Deu-net: Ddeformable U-Net for 3D cardiac mri video segmentation[C]// Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference. Lima: MICCAI, 2020: 98-107. [32] Zheng Qiao, Delingette H, Duchateau N, et al. 3-D consistent and robust segmentation of cardiac images by deep learning with spatial propagation[J]. IEEE Transactions on Medical Imaging, 2018, 37:2137-2148. [33] 徐佳陈, 肖志勇. 心脏动态 MRI 图像分割的时空多尺度网络[J]. 中国图象图形学报, 2022, 27(3):862-872. [34] 尹慧平, 张耀楠, 何颖. 基于 CT 心脏图像的腔体区域分割新算法[J]. 计算机系统应用, 2017:26(11):292-295. [35] 刘畅, 林楠, 曹仰杰,等. Seg-capnet: 心脏 MRI 图像分割神经网络模型[J]. 中国图象图形学报, 2021, 26(2): 452-463. [36] Liang Jiajun, Pan Huijuan, Xiang Zhuo, et al. Echocardiographic segmentation based on semi-supervised deep learning with attention mechanism[J]. Multimedia Tools and Applications, 2024, 83(12): 36953-36973.
