Abstract：In minimally invasive surgery, the smoke generated by operations such as electrocautery and laser ablation seriously fade image quality, not only obstructs the doctor’s field of view and increases the risk of surgery, but also reduces the performance of computer-assisted surgery algorithms (such as segmentation, 3D reconstruction, tracking, etc.). Therefore, the smoke needs to be removed real time to maintain a clear vision. This paper proposed a desmoking algorithm based on the improved U-Net network. To retain more image details, we add images which undergone Laplace pyramid transformation to the encoder part; and to improve network's performance, we add attention mechanism module (CBAM) to decoder part. The laparoscopic image was provided by the Hamlyn Center and used as the original dataset (15000 training images, 1000 synthetic smoke test images, and 129 real smoke test images), and the Blender software was used to simulate various situations of smoke which was added to the laparoscopic image, and the composite image was obtained and sent to the improved U-Net model for training, and performed 5-fold cross-validation. We obtained a PSNR value of 31.05 and SSIM index of 0.98 on the composed dataset. These two indicators showed that the smoke-purified image was very similar to the original image, which helped restore the real vision of the human body during surgery. The average running time is 90.91 fps, which is applicable in a real-time medical system. The obtained results are better than the other six methods based on physical or based on GAN, therefore the proposed approach provided a high-quality solution for the endoscopic smoke removal algorithm, which helps doctors get a clear surgical field of vision.
林金朝, 蒋媚秋, 庞宇, 王慧倩. 基于改进U-Net网络的内窥镜图像烟雾净化算法[J]. 中国生物医学工程学报, 2021, 40(3): 291-300.
Lin Jinzhao, Jiang Meiqiu, Pang Yu, Wang Huiqian. A Desmoking Algorithm for Endoscopic Images Based on Improved U-Net. Chinese Journal of Biomedical Engineering, 2021, 40(3): 291-300.
[1] 徐忠,刘洪英,皮喜田. 医用超细内窥镜系统研究 [J]. 中国生物医学工程学报, 2014, 33(1):107-111. [2] 范姗慧,刘士臣,曹鹗. 无线胶囊内窥镜图像小肠息肉的自动识别 [J]. 中国生物医学工程学报, 2019, 38(5):522-532. [3] He Kaiming, Sun Jian, Tang Xiaoou et al. Single image haze removal using dark channel prior [J]. IEEE Transactions on Pattern Analysis and machine Intelligence, 2011, 33(12): 2341-2353. [4] Tchaka K, Pawar VM, Stoyanov D. Chromaticity based smoke removal in endoscopic images [C]//Medical Imaging 2017: Image Processing. 2017:101331M.1-101331M.8. [5] Ancuti CO, Ancuti C. Single image dehazing by multi-scale fusion [J]. IEEE Transactions on Image Processing, 2013, 22(8): 3271-3282. [6] Baid A, Kotwal A, Bhalodia R, et al. Joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and Bayesian inference[C]//2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). Melbourne: IEEE Press, 2017: 732-736. [7] Kotwal A, Bhalodia R, Awate SP. Joint desmoking and denoising of laparoscopy images[C] //2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI 2016). Prague: IEEE Press, 2016, 1050-1054. [8] Luo X, Mcleod AJ, Pautler SE, et al. Vision-based surgical field defogging [J]. IEEE Transactions on Medical Imaging, 2017, 36(10): 2021-2030. [9] Ancuti CO, Ancuti C. Single image dehazing by multi-scale fusion [J]. IEEE Transactions on Image Processing, 2013, 22(8): 3271-3282. [10] Cai Bolun, Xu Xiangmin, Jia Kui, et al. Dehazenet: An end-to-end system for single image haze removal [J]. IEEE Transactions on Image Processing, 2016, 25(11): 5187-5198. [11] Yang Dong, Sun Jian, Proximal. dehaze-net: A prior learning-based deep network for single image dehazing [C] //Proceedings of the European Conference on Computer Vision (ECCV). Munich: IEEE Press, 2018: 702-717. [12] Salazarcolores S, Albertomoreno H, Flores G, et al. Laparoscopy surgery CO2 removal via generative adversary network and dark channel prior [EB/OL]. arxiv.org/abs/1909.12314, 2019-09-26/ 2020-09-02. [13] Chen Long, Tang Wen, John NW. Unsupervised learning of surgical smoke removal from simulation [J]. Computer Assisted Surgery, 2018, 123(10): 3873-3879. [14] Godard C, Aodha OM, Brostow GJ, et al. Unsupervised monocular depth estimation with left-right consistency[C] //Computer Vision and Pattern Recognition. Hawaii: IEEE Press, 2017: 6602-6611. [15] Wang Congcong, Mohammed AK, Cheikh FA, et al. Multiscale deep desmoking for laparoscopic surgery[C]// Image Processing. Taipei: IEEE Press, 2019:1117-1126. [16] Bolkar S, Wang Congcong, Cheikh FA, et al. Deep smoke removal from minimally invasive surgery videos[C] //2018 IEEE International Conference on Image Processing. Athens: IEEE Press, 2018: 3403-3407. [17] Sidorov O, Wang Congcong, Cheikh FA, et al. Generative smoke removal[J]. Computer Vision and Pattern Recognition, 2019, 116:81-92. [18] Chen Long, Tang Wen, John NW, et al. De-smokeGCN: Generative cooperative networks for joint surgical smoke detection and removal [J]. IEEE Transactions on Medical Imaging, 2019, 39(5): 1615-1625. [19] Venkatesh V, Sharma N, Srivastava V, et al. Unsupervised smoke to desmoked laparoscopic surgery images using contrast driven Cyclic-DesmokeGAN [J]. Computers in Biology and Medicine, 2020, 123:1038-1046. [20] Salazar-Colores, Sebastián H. Desmoking laparoscopy surgery images using an image-to-image translation guided by an embedded dark channel[EB/OL].arxiv.org/abs/2004.08947,2020-04-19/2020-09-20. [21] Ronneberger O, Fischer P, Brox T, et al. U-Net: Convolutional networks for biomedical image segmentation [EB/OL] //lmb.informatik.uni-freiburg.de/people/ronneber/u-net/,2015/2020-09-02. [22] Skinner KA, Zhang J, Olson EA, et al. UWStereoNet: Unsupervised learning for depth estimation and color correction of underwater stereo imagery[C]//International Conference on Robotics and Automation. Montreal: IEEE Press, 2019: 7947-7954. [23] 陈浩,王延杰. 基于拉普拉斯金字塔变换的图像融合算法研究[J]. 激光与红外, 2009, 39(4):439-442. [24] Woo S, Park J, Lee J, et al. CBAM: Convolutional block attention module[C]//European Conference on Computer Vision. Berlin: Springer-Verlag, 2018: 3-19. [25] Shin J, Kim M, Paik J, et al. Radiance-reflectance combined optimization and structure-guided l0-norm for single image dehazing [J]. IEEE Transactions on Multimedia, 2019, 22(1):1-16. [26] Isola P, Zhu Junyan, Zhou Tinghui, et al. Image-to-image translation with conditional adversarial networks[C]//Computer Vision and Pattern Recognition. Hawaii: IEEE Press, 2017: 5967-5976. [27] Zhuang Liu, Mingjie Sun, Tinghui Zhou et al. Rethinking the value of network pruning [EB/OL]. arxiv.org/abs/1810.05270, 2019-10-11/2020-09-02.
