Deep Learning Methods for Image Analysis and Synthesis for Intensity Modulated Radiotherapy:a Review
Liu Guocai1,2#*, Gu Dongdong1, Liu Xiao1, Liu Jinguang1, Liu Yanfei1, Zhang Maodan1
1(College of Electrical and Information Engineering, Hunan University, Changsha 410082, China) 2(National EngineeringResearch Center for Robot Visual Perception and Control Technology, Changsha 410082, China)
Abstract：Cancer is a common problem that seriously threatens human health. 60% to 70% of cancer patients need radiotherapy. Currently intensity modulated radiotherapy (IMRT) is one main radiotherapy technique that is widely applied in clinics. This paper reviewed deep learning methods, key technologies and future directions for IMRT, including clinical CT/CBCT/MRI/PET-guided IMRT technologies, and supervised or unsupervised deep convolutional neural networks or generative adversarial networks for the segmentation, registration and image-to-image translation of CT/CBCT/MRI/PET images of tumors.
刘国才, 顾冬冬, 刘骁, 刘劲光, 刘焰飞, 张毛蛋. 用于肿瘤调强放射治疗影像分析与转换的深度学习方法[J]. 中国生物医学工程学报, 2022, 41(2): 224-237.
Liu Guocai, Gu Dongdong, Liu Xiao, Liu Jinguang, Liu Yanfei, Zhang Maodan. Deep Learning Methods for Image Analysis and Synthesis for Intensity Modulated Radiotherapy:a Review. Chinese Journal of Biomedical Engineering, 2022, 41(2): 224-237.
[1] 郑荣寿, 孙可欣, 张思维, 等. 2015年中国恶性肿瘤流行情况分析 [J] . 中华肿瘤杂志, 2019, 41(1): 19-28. [2] Rebecca L, Siegel MPH, Kimberly D, et al. Cancer statistics, 2019 [J]. CA: A Cancer Journal for Clinicians, 2019, 69(1): 7-34. [3] Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries [J]. CA: A Cancer Journal for Clinicians, 2018, 68(6): 394-424. [4] Bernier J, Hall EJ, Giaccia A. Radiation oncology: A century of achievements [J]. Nature Reviews Cancer, 2004, 4(9): 737-747. [5] Wang Chunhao, Zhu Xiaofeng, Julian CH, et al. Artificial intelligence in radiotherapy treatment planning: present and future [J]. Technology in Cancer Research & Treatment, 2019, 18: 1-11. [6] Kosmin M, Ledsam J, Romera-Paredes B, et al. Rapid advances in auto-segmentation of organs at risk and target volumes in head and neck cancer [J]. Radiotherapy and Oncology, 2019, 135: 130-140. [7] Rigaud B, Simon A, Castelli J, et al. Deformable image registration for radiation therapy: principle, methods, applications and evaluation [J]. Acta Oncologica, 2019, 58(9): 1225-1237. [8] Lin Li, Dou Qi, Jin Yueming, et al. Deep learning for automated contouring of primary tumor volumes by MRI for nasopharyngeal carcinoma [J]. Radiology, 2019, 291(3): 677-686. [9] Yang Jinzhong, Veeraraghavan H, Armato III S G, et al. Auto segmentation for thoracic radiation treatment planning: A grand challenge at AAPM 2017 [J]. Medical physics, 2018, 45(10): 4568-4581. [10] Li Hongsheng. Automatic structure segmentation for radiotherapy planning challenge 2019 [DB/OL]. https://structseg2019.grand-challenge.org/Home/, 2019-12-26/2021-11-15. [11] 田娟秀,刘国才,谷珊珊,等. 基于3D深度残差全卷积网络的头颈CT放疗危及器官自动勾画 [J]. 中国生物医学工程学报, 2019, 38(3): 257-265. [12] Zhang Maodan, Liu Guocai, Tian Juanxiu, et al. Improved U-Net with multi-scale cross connection and dilated convolution for brain tumor segmentation [C] // 2019 International Conference on Medical Imaging Physics and Engineering. Shenzhen: IEEE, 2019: 1-5. [13] Sahiner B, Pezeshk A, Hadjiiski LM, et al. Deep learning in medical imaging and radiation therapy [J]. Medical Physics, 2019, 46(1): e1-e36. [14] Yi X, Walia E, Babyn P. Generative adversarial network in medical imaging: a review [J]. Medical Image Analysis, 2019, 58: 101552. [15] 田娟秀, 刘国才, 谷珊珊, 等. 医学图像分析深度学习方法研究与挑战 [J]. 自动化学报,2018, 44(3): 401-424. [16] Jue J, Jason H, Neelam T, et al. Integrating cross-modality hallucinated MRI with CT to aid mediastinal lung tumor segmentation [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 221-229. [17] Girum KB, Créhange G, Hussain R, et al. Deep generative model-driven multimodal prostate segmentation in radiotherapy [C] //Workshop on Artificial Intelligence in Radiation Therapy. Shenzhen: Springer-Cham, 2019: 119-127. [18] Jia Xiaoqian, Wang Sicheng, Liang Xiao, et al. Cone-beam computed tomography (CBCT) segmentation by adversarial learning domain adaptation [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 567-575. [19] Yuan Wenguang, Wei Jia, Wang Jiabing, et al. Unified attentional generative adversarial network for brain tumor segmentation from multimodal unpaired images [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 229-237. [20] Huang Jiabin, Zhuo Enhong, Li Haojiang, et al. Achieving accurate segmentation of nasopharyngeal carcinoma in MR images through recurrent attention [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 494-502. [21] Jin Dakai, Guo Dazhou, Ho T-Y, et al. Deep esophageal clinical target volume delineation using encoded 3D spatial context of tumors, lymph nodes, and organs at risk [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 603-612. [22] Men Kuo, Chen Xinyuan, Zhang Yw, et al. Deep deconvolutional neural network for target segmentation of nasopharyngeal cancer in planning computed tomography images [J].Frontiers in Oncology, 2017, 7: 315-324. [23] Chen Cheng, Dou Qi, Jin Yueming, et al. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 447-456. [24] Huang Yijie, Dou Qi, Wang Zixian, et al. 3D RoI-aware U-Net for accurate and efficient colorectal tumor segmentation [J]. IEEE Transactions on Cybernetics, 2020, 1: 1-12. [25] Mirza M, Osindero S. Conditional Generative Adversarial Nets [J]. Computer Science, 2014, 2014: 2672-2680. [26] Zhu JY, Park T, Isola P, et al. Unpaired image-to image translation using cycle-consistent adversarial networks [C] // Proceedings of the IEEE International Conference on Computer Vision. Cambridge: IEEE, 2017: 2242-2251. [27] Chen S, Bortsova G, Juárez AGU, et al. Multi-task attention-based semi-supervised learning for medical image segmentation [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 457-465. [28] Harrison AP, Xu Z, George K, et al. Progressive and multi-path holistically nested neural networks for pathological lung segmentation from CT images [C] //International Conference on Medical Image Computing and Computer-Assisted Intervention. Quebec: Springer-Cham, 2017: 621-629. [29] Elmahdy MS, Wolterink JM, Sokooti H, et al. Adversarial optimization for joint registration and segmentation in prostate CT radiotherapy [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 366-374. [30] Lei Yang, Fu Yabo, Harms J, et al. 4D-CT deformable image registration using an unsupervised deep convolutional neural network [C] // Workshop on Artificial Intelligence in Radiation Therapy. Shenzhen: Springer-Cham, 2019: 26-33. [31] Hu Yipeng, Modat M, Gibson E, et al. Label-driven weakly-supervised learning for multimodal deformable image registration [C] // 2018 IEEE 15th International Symposium on Biomedical Imaging. Washington, DC: IEEE, 2018: 1070-1074. [32] Hu Yipeng, Modat M, Gibson E, et al. Weakly-supervised convolutional neural networks for multimodal image registration [J]. Med Image Anal, 2018, 49: 1-13. [33] Hu Yipeng, Gibson E, Barratt DC, et al. Conditional segmentation in lieu of image registration [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer-Cham, 2019: 401-409. [34] Mohamed S. Thyrza Jagt E, Zinkstok RT, et al. Robust contour propagation using deep learning and image registration for online adaptive proton therapy of prostate cancer [J]. Med Phys, 2019, 46 (8): 3329-3343. [35] Kearney V, Haaf S, Sudhyadhom A, et al. An unsupervised convolutional neural network-based algorithm for deformable image registration [J]. Phys Med Biol, 2018, 63(18): 185017. [36] Gu Dongdong, Cao Xiaohuan, Ma Shanshan, et al. Pair-wise and group-wise deformation consistency in deep registration network [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Lima: Springer-Cham, 2020: 171-180. [37] Gu Dongdong, Liu Guocai, Cao Xioahuan, et al. A consistent deep registration network with group data modeling [J]. Computerized Medical Imaging and Graphics, 2021, 90: 101904. [38] Gu Dongdong, Cao Xioahuan, Liu Guocai, et al. Variational encoding and decoding for hybrid supervision of registration network [C] //International Workshop on Machine Learning in Medical Imaging, Oct, 2021 [LNCS ahead of print]. [39] Gu Dongdong, Liu Guocai, Tian Juanxiu, et al. Two-stage unsupervised learning method for affine and deformable medical image registration [C] // 2019 IEEE International Conference on Image Processing. Taipei: IEEE, 2019: 1332-1336. [40] Edmund JM, Nyholm T. A review of substitute CT generation for MRI-only radiation therapy [J]. Radiation Oncology, 2017, 12(1): 28. [41] Maspero M, Savenije MH, Dinkla AM, et al. Dose evaluation of fast synthetic-CT generation using a generative adversarial network for general pelvis MR-only radiotherapy [J]. Physics in Medicine & Biology, 2018, 63(18): 185001. [42] Nie Dong, Cao Xiaohuan, Gao Yaozong, et al. Estimating CT image from MRI data using 3D fully convolutional networks [C] //Deep Learning and Data Labeling for Medical Applications. DLMIA 2016, LABELS 2016. Athens: Springer-Cham, 2016: 170-178. [43] Emami H, Dong M, Nejad-Davarani SP, et al. Generating synthetic CTs from magnetic resonance images using generative adversarial networks [J]. Medical Physics, 2018, 45(8): 3627-3636. [44] Karim A, Chenming J, Marc F, et al. MedGAN: Medical image translation using GANs [J]. Computerized Medical Imaging and Graphics, 2020, 79: 101684. [45] Wu H, Jiang X, Jia F. UC-GAN for MR to CT image synthesis [C] // Artificial Intelligence in Radiation Therapy. Shenzhen: Springer-Cham, 2019: 146-153. [46] Yang H, Jian S, Carass A, et al. Unpaired brain MR-to-CT synthesis using a structure-constrained CycleGAN [C] // Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Granada: Springer-Cham, 2018: 174-182. [47] Prokopenko D, Stadelmann JV, Schulz H, et al. Unpaired synthetic image generation in radiology using GANs [C] // Workshop on Artificial Intelligence in Radiation Therapy. Shenzhen: Springer-Cham, 2019:94-101. [48] Peng Yinglin, Chen Shupeng, Qin An, et al. Magnetic resonance-based synthetic computed tomography images generated using generative adversarial networks for nasopharyngeal carcinoma radiotherapy treatment planning [J]. Radiotherapy and Oncology, 2020, 150: 217-224. [49] Hemsley M, Chugh B, Ruschin M, et al. Deep generative model for synthetic-CT generation with uncertainty predictions [C] // International Conference on Medical Image Computing and Computer-Assisted Intervention. Lima: Springer-Cham, 2020: 834-844. [50] Li Yafei, Li Wen, Xiong Jing, et al. Comparison of supervised and unsupervised deep learning methods for medical image synthesis between computed tomography and magnetic resonance images [J]. BioMed Research International, 2020, 2020: 5193707. [51] Zhang Zizhao, Yang Lin, Zheng Yefeng. Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network [C] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9242-9251. [52] Han Xiao. MR-based synthetic CT generation using a deep convolutional neural network method [J]. Medical physics, 2017, 44(4): 1408-1419. [53] Xiang Lei, Wang Qian, Nie Dong, et al. Deep embedding convolutional neural network for synthesizing CT image from T1-weighted MR image [J]. Med Image Anal, 2018, 47: 31-44. [54] Dinkla AM, Florkow MC, Maspero M, et al. Dosimetric evaluation of synthetic CT for head and neck radiotherapy generated by a patch-based three-dimensional convolutional neural network [J]. Medical Physics, 2019, 46(9): 4095-4104. [55] Wu Haitao, Jiang Xiling, Jia Fucang. UC-GAN for MR to CT image synthesis [C] // Workshop on Artificial Intelligence in Radiation Therapy. Shenzhen: Springer-Cham, 2019: 146-153. [56] Baydoun A, Xu K, Jin UH, et al. Synthetic CT generation of the pelvis in patients with cervical cancer: a single input approach using generative adversarial network [J]. IEEE Access, 2021, 9:17208-17221. [57] Qi Mengke, Li Yongbao, Wu Aiqian, et al. Multi-sequence MR image-based synthetic CT generation using a generative adversarial network for head and neck MRI-only radiotherapy [J]. Medical Physics, 2020, 47(4): 1880-1894. [58] Tie Xin, Lam S, Zhang Yong, et al. Pseudo-CT generation from multi-parametric MRI using a novel multi-channel multi-path conditional generative adversarial network for nasopharyngeal carcinoma patients [J]. Medical Physics, 2020, 47(4): 1750-1762. [59] Chen Shupeng, Qin An, Zhou Dingyi, et al. Technical note: U-net-generated synthetic CT images for magnetic resonance imaging-only prostate intensity-modulated radiation therapy treatment planning [J]. Medical Physics, 2018, 45(12): 5659-5665. [60] Neppl S, Landry G, Kurz C, et al. Evaluation of proton and photon dose distributions recalculated on 2D and 3D Unet-generated pseudo CTs from T1-weighted MR head scans [J]. Acta Oncologica, 2019, 58(10): 1429-1434. [61] Largent A, Barateau A, Nunes JC, et al. Comparison of deep learning-based and patch-based methods for pseudo-CT generation in MRI-based prostate dose planning [J]. International Journal of Radiation Oncology Biology Physics, 2019, 105(5):1137-1150. [62] Low DA, Harms WB, Mutic S, Purdy JA. A technique for the quantitative evaluation of dose distributions [J]. Med Phys, 1998, 25:656-661.
