|
|
|
| Multi-FHRNet: Multimodal Fetal Diagnosis Based on Image and Text with Adaptive Weighting Fusion |
| Zhang Ling1, Zhao Zhidong2*, Zhang Yiefei2#, Jiao Pengfei2, Deng Yanjun2, Zhang Xianfei2 |
1(School of Communication Engineering, Hangzhou Dianzi University, Hangzhou 310018, China)
2(School of Cyberspace, Hangzhou Dianzi University, Hangzhou 310018, China) |
|
|
|
|
Abstract Distress or acidosis resulted from hypoxia can lead to irreversible consequences such as fetal organ damage even death. Intelligent cardiotocography (ICTG) is an important tool for monitoring fetal health by continuously and synchronously recording fetal heart rate (FHR) signals in late pregnancy. The existing FHR-based unimodal ICTG combined with machine learning algorithms for assisted diagnosis method neglects the complementarity between different modalities data of FHR and the influence of clinical physiological factors on the fetus. In this study, we proposed a multimodal learning model with adaptive weighted fusion of images and text. In particular, we designed a multimodal feature extraction network consisting of a Vision Transformer (ViT) image encoder and a convolutional neural networks (CNN) text encoder. Adaptive weighting fusion (AWF) was proposed to fuse multimodal features, instead of direct concatenates. Considering the impact of realistic risk factors, textual data can be constructed by combining the clinical data and the morphological features of FHR signals. Meanwhile, the Markov transfer field was employed to convert signals into images as complementary data. Using 200 sets of public clinical real-world FHR signals, multiple performance comparisons, parameter optimization, and ablation experiments were carried out. The experiment results indicated that the Multi-FHRNet outperformed traditional unimodal ICTG methods, withthe highest accuracy, precision, recall, and F1-score of 96.02%, 93.10%, 99.29%, 95.45% and 93.48%, respectively. The algorithm in this paper may help to detect and treat abnormal fetuses during labor.
|
|
Received: 05 November 2024
|
|
|
|
|
|
[1] Glick I, Kadish E, Rottenstreich M.Management of pregnancy in women of advanced maternal age: improving outcomes for mother and baby[J].International Journal of Women′s Health, 2021,13: 751-759.
[2] Alfirevic Z, Gyte GML, Cuthbert A, et al.Continuous cardiotocography (CTG) as a form of electronic fetal monitoring (EFM) for fetal assessment during labour[J].Cochrane Database of Systematic Reviews, 2013, (5):CD006066.
[3] Palomäki O, Luukkaala T, Luoto R, et al.Intrapartum cardiotocography-the dilemma of interpretational variation[J].Journal of Perinatal Medicine, 2006, 34(4): 298-302.
[4] Ayres-de-Campos D, Spong CY, Chandraharan E.FIGO consensus guidelines on intrapartum fetal monitoring: Cardiotocography[J].International Journal of Gynecology & Obstetrics, 2015, 131(1): 13-24.
[5] Muhammad Hussain N, Rehman AU, Othman MTB, et al.Accessing artificial intelligence for fetus health status using hybrid deep learning algorithm (AlexNet-SVM) on cardiotocographic data[J].Sensors, 2022, 22(14): 5103.
[6] Chen Yandi, Guo Ao, Chen Qinqun, et al.Intelligent classification of antepartum cardiotocography model based on deep forest[J].Biomedical Signal Processing and Control, 2021, 67: 102555.
[7] Salini Y, Mohanty SN, Ramesh JVN, et al.Cardiotocography data analysis for fetal health classification using machine learning models[J].IEEE Access, 2024,12:26005-26022.
[8] Kapila R, Saleti S.Optimizing fetal health prediction: ensemble modeling with fusion of feature selection and extraction techniques for cardiotocography data[J].Computational Biology and Chemistry, 2023, 107: 107973.
[9] Zhao Zhidong, Zhang Yefei, Deng Yanjun.A comprehensive feature analysis of the fetal heart rate signal for the intelligent assessment of fetal state[J].Journal of Clinical Medicine, 2018, 7(8): 223.
[10] Liu Mujun, Lu Yaosheng, Long Shun, et al.An attention-based CNN-BiLSTM hybrid neural network enhanced with features of discrete wavelet transformation for fetal acidosis classification[J].Expert Systems with Applications, 2021, 186: 115714.
[11] Improta G, Ricciardi C, Amato F, et al.Efficacy of machine learning in predicting the kind of delivery by cardiotocography[C]//XV Mediterranean Conference on Medical and Biological Engineering and Computing-MEDICON 2019.Coimbra: Springer International Publishing, 2020: 793-799.
[12] Intan PRD, Masum MA, Alfiany N, et al.Ensemble learning versus deep learning for Hypoxia detection in CTG signal[C]//2019 International Workshop on Big Data and Information Security (IWBIS).Bali: IEEE, 2019: 57-62.
[13] Zhao Zhidong, Deng Yanjun, Zhang Yefei, et al.DeepFHR: intelligent prediction of fetal Acidemia using fetal heart rate signals based on convolutional neural network[J].BMC Medical Informatics and Decision Making, 2019, 19(286): 1-15.
[14] Xiang Junhong, Yang Wanrong, Zhang Hua, et al.Digital signal extraction approach for cardiotocography image[J].Computer Methods and Programs in Biomedicine, 2022, 225: 107089.
[15] Fergus P, Chalmers C, Montanez CC, et al.Modelling segmented cardiotocography time-series signals using one-dimensional convolutional neural networks for the early detection of abnormal birth outcomes[J].IEEE Transactions on Emerging Topics in Computational Intelligence, 2020, 5(6): 882-892.
[16] Daydulo YD, Thamineni BL, Dasari HK, et al.Deep learning based fetal distress detection from time frequency representation of cardiotocogram signal using Morse wavelet: research study[J].BMC Medical Informatics and Decision Making, 2022, 22(1): 329.
[17] Cao Zhen, Wang Guoqiang, Xu Ling, et al.Intelligent antepartum fetal monitoring via deep learning and fusion of cardiotocographic signals and clinical data[J].Health Information Science and Systems, 2023, 11(1): 16.
[18] Baltrušaitis T, Ahuja C, Morency LP.Multimodal machine learning: A survey and taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(2): 423-443.
[19] Huang SC, Shen Liyue, Lungren MP, et al.Gloria: a multimodal global-local representation learning framework for label-efficient medical image recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal: IEEE, 2021: 3942-3951.
[20] Ding Zhaisheng, Li Haiyan, Guo Yi, et al.M4fnet: multimodal medical image fusion network via multi-receptive-field and multi-scale feature integration[J].Computers in Biology and Medicine, 2023, 159: 106923.
[21] Petrozziello A, Redman CWG, Papageorghiou AT, et al.Multimodal convolutional neural networks to detect fetal compromise during labor and delivery[J].IEEE Access, 2019, 7: 112026-112036.
[22] Asfaw D, Jordanov I, Impey L, et al.Multimodal deep learning for predicting adverse birth outcomes based on early labour data[J].Bioengineering, 2023, 10(6): 730.
[23] Fei Yue, Chen Fan, He Lifang, et al.Intelligent classification of antenatal cardiotocography signals via multimodal bidirectional gated recurrent units[J].Biomedical Signal Processing and Control, 2022, 78: 104008.
[24] Zhang Yefei, Deng Yanjun, Zhou Zhixin, et al.Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework[J].Frontiers in Physiology, 2022, 13: 1021400.
[25] Zhao Zhidong, Zhu Jiawei, Jiao Pengfei, et al.Hybrid-FHR: a multi-modal AI approach for automated fetal acidosis diagnosis[J].BMC Medical Informatics and Decision Making, 2024, 24(1): 19.
[26] Wang Zhiguang, Oates T.Imaging time-series to improve classification and imputation[J].Computer Science, 1 Jun, 2015[Epub ahead of print].
[27] Warmerdam GJJ, Vullings R, Bergmans JWM, et al.Reliability of spectral analysis of fetal heart rate variability[C]//2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.Chicago: IEEE, 2014: 2817-2820.
[28] Dosovitskiy A.An image is worth 16×16 words: Transformers for image recognition at scale[J].Computer Science, 22 Oct, 2020[Epub ahead of print].
[29] 周腊珍,陈红池,李秋霞,等.基于Transformer深度学习模型在医学图像分割中的研究进展[J].中国生物医学工程学报,2024,43(4):467-476.
[30] Liang Huanwen, Lu Yu.A CNN-RNN unified framework for intrapartum cardiotocograph classification[J].Computer Methods and Programs in Biomedicine, 2023, 229: 107300.
[31] Alsaggaf W, Cömert Z, Nour M, et al.Predicting fetal hypoxia using common spatial pattern and machine learning from cardiotocography signals[J].Applied Acoustics, 2020, 167: 107429. |
|
|
|
