A Review on Intelligent Detection Methods of Gastrointestinal Tract Lesions Based on WirelessCapsule Endoscopy Images
Fan Shanhui1, Wei Shangguang1, Wei Kaihua1, Fan Yihong2, Lv Bin2, Fan Kai1, Li Lihua1*
1(School of Automation (School of Artificial Intelligence), Hangzhou Dianzi University, Hangzhou 310018, China) 2(Department of Gastroenterology, Zhejiang Provincial Hospital of Traditional Chinese Medicine, Hangzhou 310006, China)
Abstract：Wireless capsule endoscopy (WCE) is a non-invasive and painless method for gastrointestinal (GI) tract examination, but it produce tens of thousands of images during the examination, which may only contain a few abnormal images. The review work highly relies on the clinical experience of physicians, which is time-consuming and easily cause false detection and/or missed detection. Therefore, exploring automated detection methods of WCE abnormal images to aid to clinical diagnosis with high accuracy and efficiency has become a research hotspot in this field. This paper reviewed recent researches on intelligently detecting GI lesions on WCE images. We first introduced the basic principles and evaluation metrics of intelligent detection methods, and then outlined the researches on intelligent detection methods of WCE abnormal images in recent years from the perspectives of traditional machine learning and deep learning algorithms, and summarized the advantages and shortcomings of the reported methods. Finally, this paper discussed and concluded the current challenges and possible future research directions in automatic lesion classification of WCE.
范姗慧, 韦尚光, 魏凯华, 范一宏, 吕宾, 樊凯, 厉力华. 基于无线胶囊内窥镜图像的胃肠道病变智能检测方法综述[J]. 中国生物医学工程学报, 2024, 43(5): 596-608.
Fan Shanhui, Wei Shangguang, Wei Kaihua, Fan Yihong, Lv Bin, Fan Kai, Li Lihua. A Review on Intelligent Detection Methods of Gastrointestinal Tract Lesions Based on WirelessCapsule Endoscopy Images. Chinese Journal of Biomedical Engineering, 2024, 43(5): 596-608.
[1] Iddan G, Meron G, Glukhovsky A, et al. Wireless capsule endoscopy [J]. Nature, 2000, 405(6785): 417-417. [2] Rahim T, Usman M, Shin S. A survey on contemporary computer-aided tumor, polyp, and ulcer detection methods in wireless capsule endoscopy imaging [J]. Computerized Medical Imaging and Graphics, 2020, 85: 101767. [3] Maieron A, Hubner D, Blaha B, et al. Multicenter retrospective evaluation of capsule endoscopy in clinical routine [J]. Endoscopy, 2004, 36(10): 864-868. [4] Zheng Yuanpu, Hawkins L, Wolff J, et al. Detection of lesions during capsule endoscopy: physician performance is disappointing [J]. Official Journal of the American College of Gastroenterology | ACG, 2012, 107(4): 554-560. [5] Muhammad K, Khan S, Kumar N, et al. Vision-based personalized wireless capsule endoscopy for smart healthcare: taxonomy, literature review, opportunities and challenges [J]. Future Generation Computer Systems, 2020, 113: 266-280. [6] Haefner M, Brunauer L, Payer H, et al. Computer-aided classification of zoom-endoscopical images using fourier filters [J]. IEEE Transactions on Information Technology in Biomedicine, 2010, 14(4): 958-970. [7] 周飞燕, 金林鹏, 董军. 卷积神经网络研究综述 [J]. 计算机学报, 2017, 40(6): 1229-1251. [8] Lecun Y, Bengio Y, Hinton G. Deep learning [J]. Nature, 2015, 521(7553): 436-444. [9] Cui Lei, Hu Chao, Zou Yuexian, et al. Bleeding detection in wireless capsule endoscopy images by support vector classifier [C]// The 2010 IEEE International Conference on Information and Automation. Harbin: IEEE, 2010: 1746-1751. [10] Fu Yanan, Zhang Wei, Mandal M., et al. Computer-aided bleeding detection in wce video [J]. IEEE Journal of Biomedical and Health Informatics, 2014, 18(2): 636-642. [11] Yuan Yixuan, Li Baopu, Meng Max Q-H. Bleeding frame and region detection in the wireless capsule endoscopy video [J]. IEEE Journal of Biomedical and Health Informatics, 2016, 20(2): 624-630. [12] Ghosh T, Fattah S, Wahid K. Chobs: Color histogram of block statistics for automatic bleeding detection in wireless capsule endoscopy video [J]. IEEE Journal of Translational Engineering in Health and Medicine, 2018, 6: 1-12. [13] 范姗慧, 刘士臣, 曹鹗,等. 无线胶囊内窥镜图像小肠息肉的自动识别 [J]. 中国生物医学工程学报, 2019, 38(5): 522-532. [14] Li Baopu, Meng Max Q-H. Automatic polyp detection for wireless capsule endoscopy images [J]. Expert Systems with Applications, 2012, 39(12): 10952-10958. [15] Yuan Yixuan, Meng Max Q-H. A novel feature for polyp detection in wireless capsule endoscopy images [C]// 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. Chicago: IEEE, 2014: 5010-5015. [16] Yuan Yixuan, Li Baopu, Meng Max QH. Improved bag of feature for automatic polyp detection in wireless capsule endoscopy images [J]. IEEE Transactions on Automation Science and Engineering, 2016, 13(2): 529-535. [17] Yang Jianjun, Chang Liping, Li Sheng, et al. WCE polyp detection based on novel feature descriptor with normalized variance locality-constrained linear coding [J]. International Journal of Computer Assisted Radiology and Surgery, 2020, 15: 1291-1302. [18] Li Baopu, Meng Max QH. Texture analysis for ulcer detection in capsule endoscopy images [J]. Image and Vision computing, 2009, 27(9): 1336-1342. [19] Yuan Yixuan, Wang Jiaole, Li Baopu, et al. Saliency based ulcer detection for wireless capsule endoscopy diagnosis [J]. IEEE Transactions on Medical Imaging, 2015, 34(10): 2046-2057. [20] Suman S, Hussin F, Malik A, et al. Feature selection and classification of ulcerated lesions using statistical analysis for WCE images [J]. Applied Sciences, 2017, 7(10): 1097. [21] Souaidi M, Ansari M. Multi-scale analysis of ulcer disease detection from wce images [J]. IET Image Processing, 2019, 13(12): 2233-2244. [22] Eid A, Charisis V, Hadjileontiadis L, et al. A curvelet-based lacunarity approach for ulcer detection from wireless capsule endoscopy images [C]// Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems. Porto: IEEE, 2013: 273-278. [23] Karargyris A, Bourbakis N. Detection of small bowel polyps and ulcers in wireless capsule endoscopy videos [J]. IEEE Transactions on biomedical engineering, 2011, 58(10): 2777-2786. [24] Yuan Yixuan, Li Baopu, Meng Max QH. WCE abnormality detection based on saliency and adaptive locality-constrained linear coding [J]. IEEE Transactions on Automation Science and Engineering, 2016, 14(1): 149-159. [25] Charfi S, Ansari M. Computer-aided diagnosis system for colon abnormalities detection in wireless capsule endoscopy images [J]. Multimedia Tools and Applications, 2018, 77(3): 4047-4064. [26] Jani K, Srivastava R, Srivastava S. Computer aided medical image analysis for capsule endoscopy using multi-class classifier [C]// 2019 IEEE 5th International Conference for Convergence in Technology (I2CT). Bombay: IEEE, 2019: 1-5. [27] Iakovidis D, Koulaouzidis A. Software for enhanced video capsule endoscopy: challenges for essential progress [J]. Nature Reviews Gastroenterology & Hepatology, 2015, 12(3): 172-186. [28] Krizhevsky A, Sutskever L, Hinton G. Imagenet classification with deep convolutional neural networks [J]. Communications of the ACM, 2017, 60(6): 84-90. [29] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [EB/OL]. https://arxiv.org/abs/1409.1556, 2015-04-10/2023-04-12. [30] Szegedy C, Liu Wei, Jia Yangqing, et al. Going deeper with convolutions [C]// Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 1-9. [31] He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778. [32] Huang Gao, Liu Zhuang, Van Der Maaten L, et al. Densely connected convolutional networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4700-4708. [33] Li Panpeng, Li Ziyun, Gao Fei, et al. Convolutional neural networks for intestinal hemorrhage detection in wireless capsule endoscopy images [C]// 2017 IEEE International Conference on Multimedia and Expo (ICME). Hong Kong: IEEE, 2017: 1518-1523. [34] Fan Shanhui, Xu Lanmeng, Fan Yihong, et al. Computer-aided detection of small intestinal ulcer and erosion in wireless capsule endoscopy images [J]. Physics in Medicine & Biology, 2018, 63(16): 165001. [35] Ding Zhen, Shi Huiying, Zhang Hao, et al. Gastroenterologist-level identification of small-bowel diseases and normal variants by capsule endoscopy using a deep-learning model [J]. Gastroenterology, 2019, 157(4): 1044-1054. [36] Yogapriya J, Chandran V, Sumithra M, et al. Gastrointestinal tract disease classification from wireless endoscopy images using pretrained deep learning model [J]. Computational and Mathematical Methods in Medicine, 2021, 2021: 5940433. [37] Konstantin P, Ranheim RK, Carsten G, et al. Kvasir: a multi-class image dataset for computer aided gastrointestinal disease detection [C]// Proceedings of the 8th ACM on Multimedia Systems Conference. Taipei: ACM, 2017: 164-169. [38] Alaskar H, Hussain A, Al-Aseem N, et al. Application of convolutional neural networks for automated ulcer detection in wireless capsule endoscopy images [J]. 2019, 19(6): 1265. [39] Wang Sen, Xing Yuxiang, Zhang Li, et al. Deep convolutional neural network for ulcer recognition in wireless capsule endoscopy: experimental feasibility and optimization [J]. Computational and Mathematical Methods in Medicine, 2019, 2019: 7546215. [40] Laiz P, Vitrià J, Wenzek H, et al. WCE polyp detection with triplet based embeddings [J]. Computerized Medical Imaging and Graphics, 2020, 86: 101794. [41] Jia Xiao, Meng Max Q-H. A deep convolutional neural network for bleeding detection in wireless capsule endoscopy images [C]// 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando: IEEE, 2016: 639-642. [42] Georgakopoulos S, Iakovidis D, Vasilakakis M, et al. Weakly-supervised convolutional learning for detection of inflammatory gastrointestinal lesions [C]// 2016 IEEE International Conference on Imaging Systems and Techniques (IST). Chania: IEEE, 2016: 510-514. [43] Koulaouzidis A, Iakovidis Dimitris K, Yung Diana E, et al. KID project: an internet-based digital video atlas of capsule endoscopy for research purposes [J]. Endoscopy International Open, 2017, 05(6): E477-E483. [44] Yuan Yixuan, Meng Max QH. Deep learning for polyp recognition in wireless capsule endoscopy images [J]. Medical Physics, 2017, 44(4): 1379-1389. [45] Sadasivan V, Seelamantula C. High accuracy patch-level classification of wireless capsule endoscopy images using a convolutional neural network [C]// 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). Venice: IEEE, 2019: 96-99. [46] Mahmood S, Fareed M, Ahmed G, et al. A robust deep model for classification of peptic ulcer and other digestive tract disorders using endoscopic images [J]. Biomedicines, 2022, 10(9): 2195. [47] Smedsrud P, Thambawita V, Hicks S, et al. Kvasir-capsule, a video capsule endoscopy dataset [J]. Scientific Data, 2021, 8: 142. [48] Zhao Bailiang, Sun Wendell Q., Wang Liangchao, et al. Fusion of selected deep CNN and handcrafted features for gastritis detection from wireless capsule endoscopy images [C]// 2021 14th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). Shanghai: IEEE, 2021: 1-6. [49] Jia Xiao, Meng Max QH. Gastrointestinal bleeding detection in wireless capsule endoscopy images using handcrafted and cnn features [C]// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju: IEEE, 2017: 3154-3157. [50] Lai Zhiqiang, Jia Zhiwei. Multi-lesion classification of WCE images based on deep sparse feature selection and feature fusion [C]// 2022 3rd International Conference on Electronic Communication and Artificial Intelligence (IWECAI). Zhuhai: IEEE, 2022: 461-466. [51] Sharif M, Attique Khan M, Rashid M, et al. Deep CNN and geometric features-based gastrointestinal tract diseases detection and classification from wireless capsule endoscopy images [J]. Journal of Experimental and Theoretical Artificial Intelligence, 2021, 33(4): 577-599. [52] Jeon Y, Cho E, Moon S, et al. Deep convolutional neural network-based automated lesion detection in wireless capsule endoscopy [C]// International Forum on Medical Imaging in Asia 2019. Singapore: SPIE, 2019: 292-296. [53] Biradher S, Aparna P. Classification of capsule endoscopy images based on feature concatenation of deep neural networks [C]// 2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT). Erode: IEEE, 2021: 1-4. [54] Coelho P, Pereira A, Leite A, et al. A deep learning approach for red lesions detection in video capsule endoscopies [C]// Image Analysis and Recognition: 15th International Conference, ICIAR 2018, Póvoa de Varzim: Springer, 2018: 553-561. [55] Bouyaya D, Benierbah S. Abnormalities classification in wce images using pretrained deep learning networks [C]// The 4th International Conference on Pattern Analysis and Intelligent Systems (PAIS). Oum El Bouaghi: IEEE, 2022: 1-6. [56] Rani K, Devi G, Kumar S, et al. Classification of wireless capsule endoscopy images for bleeding using deep features fusion [C]// 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME). Maldives: IEEE, 2022: 1-6. [57] Pannu H, Ahuja S, Dang N, et al. Deep learning based image classification for intestinal hemorrhage [J]. Multimedia Tools and Applications, 2020, 79: 21941-21966. [58] Masmoudi Y, Ramzan M, Khan S, et al. Optimal feature extraction and ulcer classification from WCE image data using deep learning [J]. Soft Computing, 2022, 26(16): 7979-7992. [59] Rustam F, Siddique M, Siddiqui H, et al. Wireless capsule endoscopy bleeding images classification using cnn based model [J]. IEEE Access, 2021, 9: 33675-33688. [60] Xing Xiaohan, Yuan Yixuan, Jia Xiao, et al. A saliency-aware hybrid dense network for bleeding detection in wireless capsule endoscopy images [C]// 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). Venice: IEEE, 2019: 104-107. [61] Zhu Meilu, Chen Zhen, Yuan Yixuan. DSI-Net: deep synergistic interaction network for joint classification and segmentation with endoscope images [J]. IEEE Transactions on Medical Imaging, 2021, 40(12): 3315-3325. [62] Leenhardt R, Li C, Le Mouel J, et al. CAD-CAP: a 25,000-image database serving the development of artificial intelligence for capsule endoscopy [J]. Endoscopy International Open, 2020, 8(3): E415-E420. [63] Vallée R, De Maissin A, Coutrot A, et al. Accurate small bowel lesions detection in wireless capsule endoscopy images using deep recurrent attention neural network [C]// 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP). Kuala Lumpur: IEEE, 2019: 1-5. [64] Vallée R, De Maissin A, Coutrot A, et al. CrohnIPI: an endoscopic image database for the evaluation of automatic crohn′s disease lesions recognition algorithms [C]// Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging. Houston: SPIE, 2020: 440-446. [65] Xing Xiaohan, Yuan Yixuan, Meng Max QH. Zoom in lesions for better diagnosis: attention guided deformation network for wce image classification [J]. IEEE Transactions on Medical Imaging, 2020, 39(12): 4047-4059. [66] Guo Xiaoqing, Yuan Yixuan. Triple ANet: Adaptive abnormal-aware attention network for wce image classification [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Shenzhen: Springer, 2019: 293-301. [67] Muruganantham P, Balakrishnan S. Attention aware deep learning model for wireless capsule endoscopy lesion classification and localization [J]. Journal of Medical and Biological Engineering, 2022, 42(2): 157-168. [68] Deeba F, Islam M, Bui F, et al. Performance assessment of a bleeding detection algorithm for endoscopic video based on classifier fusion method and exhaustive feature selection [J]. Biomedical Signal Processing and Control, 2018, 40: 415-424. [69] Guo Xudong, Zhang Lulu, Hao Youguo, et al. Multiple abnormality classification in wireless capsule endoscopy images based on efficientnet using attention mechanism [J]. Review of Scientific Instruments, 2021, 92(9): 094102. [70] Alam M, Rashid R, Fattah S, et al. RAt-CapsNet: a deep learning network utilizing attention and regional information for abnormality detection in wireless capsule endoscopy [J]. IEEE Journal of Translational Engineering in Health and Medicine, 2022, 10: 3300108. [71] Vats A, Raja K, Pedersen M, et al. Multichannel residual cues for fine-grained classification in wireless capsule endoscopy [J]. IEEE Access, 2022, 10: 91414-91423. [72] Li Sheng, Cao Jing, Yao Jiafeng, et al. Adaptive aggregation with self-attention network for gastrointestinal image classification [J]. IET Image Processing, 2022, 16(9): 2384-2397. [73] Bai Long, Wang Liangyu, Chen Tong, et al. Transformer-based disease identification for small-scale imbalanced capsule endoscopy dataset [J]. Electronics, 2022, 11(17): 2747. [74] Hosain A, Islam M, Mehedi M, et al. Gastrointestinal disorder detection with a transformer based approach [C]// 2022 IEEE 13th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON). Vancouver: IEEE, 2022: 280-285. [75] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: transformers for image recognition at scale [EB/OL]. https://arxiv.org/abs/2010.11929, 2021-06-03/2023-04-12. [76] Deeba Farah, Islam Monzurul, Bui Francis M, et al. Performance assessment of a bleeding detection algorithm for endoscopic video based on classifier fusion method and exhaustive feature selection [J]. 2018, 40: 415-424. [77] 刘飞, 张俊然, 杨豪. 基于深度学习的医学图像识别研究进展 [J]. 中国生物医学工程学报, 2018, 37(1): 86-94. [78] 雷霞, 罗雄麟. 深度学习可解释性研究综述 [J]. 计算机应用, 2022, 42(11): 3588-3602.
