Wei Mengying, Li Linling, Huang Gan, Tang Fei, Zhang Zhiguo*
(School of Biomedical Engineering, Health Science Center, Shenzhen University, National Regional Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China)
Abstract：In recent years, deep learning algorithms have been developed rapidly, and they are becoming a powerful tool in biomedical engineering. Especially, there has been an increasing focus on the use of deep learning algorithms for decoding physiological, psychological or pathological states of the brain from EEG. This paper overviews current applications of deep learning algorithms in various EEG decoding tasks, and introduces commonly used algorithms, typical application scenarios, important progresses and existing problems. Firstly, we briefly describe the basic principles of deep learning algorithms used in EEG decoding, including convolutional neural network, deep belief network, auto-encoder and recurrent neural network. Then this paper discusses existing applications of deep learning on EEG, including brain-computer interfaces, cognitive neuroscience and diagnosis of brain disorders. Finally, this paper outlines some key issues that need to be addressed in future applications of deep learning for EEG decoding, such as parameter selection, computational complexity, and the capability of generalization.
[1] Sheehy N. Electroencephalography: Basic principles, clinical applications and related fields [J]. J Neurol Neurosurg Psychiatry, 1984, 47(6): 654-654. [2] Buzsaki G, Draguhn A. Neuronal oscillations in cortical networks [J]. Science, 2004, 304(5679): 1926-1929. [3] Ghshdastidar S, Adeli H, Dadmehr N. Mixed-band wavelet-chaos-neural network methodology for epilepsy and epileptic seizure detection [J]. IEEE Trans Biomed Eng, 2007, 54(9): 1545-1551. [4] Adeli H, Zhou Ziqin, Dadmehr N. Analysis of EEG records in an epileptic patient using wavelet transform [J]. J Neurosci Methods, 2003, 123(1): 69-87. [5] Charbonnier S, Zoubek L, Lesecq S, et al. Self-evaluated automatic classifier as a decision-support tool for sleep/wake staging [J]. Comput Biol Med, 2011, 41(6): 380-389. [6] Sinha RK. EEG power spectrum and neural network based sleep-hypnogram analysis for a model of heat stress [J]. J Clin Monit Comput, 2008, 22(4): 261-268. [7] McFarland DJ, Wolpaw JR. EEG-based brain-computer interfaces [J]. Curr Opin Biomed Eng, 2017, 4: 194-200. [8] McMenamin BW, Shackman AJ, Greischar LL, et al. Electromyogenic artifacts and electroencephalographic inferences revisited [J]. Neuroimage, 2011, 54(1): 4-9. [9] Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for EEG-based brain-computer interfaces: A 10-year update [J]. J Neural Eng, 2018, 15(3):031005. [10] Obermaier B, Guger C, Neuper C, et al. Hidden markov models for online classification of single trial EEG data [J]. Pattern Recognit Lett, 2001, 22(12): 1299-1309. [11] Osman AH, Alzahrani AA. New approach for automated epileptic disease diagnosis using an integrated self-organization map and radial basis function neural network algorithm[J]. IEEE Access, 2019, 7: 4741-4747. [12] Palm RB. Prediction as a candidate for learning deep hierarchical models of data [D]. Copenhagen: Technical University of Denmark, 2012. [13] Lu Na, Li Tengfei, Ren Xiaodong, et al. A deep learning scheme for motor imagery classification based on restricted boltzmann machines[J]. IEEE Trans Neural Syst Rehabil Eng, 2017, 25(6): 566-576. [14] Abdelhamid O, Mohamed A, Jiang Hui, et al. Convolutional neural networks for speech recognition [J]. IEEE Trans Audio Speech Lang Process, 2014, 22(10): 1533-1545. [15] Chen LC, Papandreou G, Kokkinos I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE Trans Pattern Anal Mach Intell, 2018, 40(4): 834-848. [16] Szegedy C, Zaremba W, Sutskever I, et al. Intriguing properties of neural networks [J]. arXiv, 2013, 1312: 6199. [17] Guo Yanming, Liu Yu, Oerlemans A, et al. Deep learning for visual understanding: A review[J]. Neurocomputing, 2016, 187: 27-48. [18] Kuremoto T, Kimura S, Kobayashi K, et al. Time series forecasting using a deep belief network with restricted Boltzmann machines [J]. Neurocomputing, 2014, 137(15):47-56. [19] Lecun Y, Bengio Y, Hinton G. Deep learning [J]. Nature, 2015, 521(7553): 436-436. [20] Li Junhua, Struzik Z, Zhang Liqing, et al. Feature learning from incomplete EEG with denoising autoencoder [J]. Neurocomputing, 2015, 165:23-31. [21] Wang Zuoyuan, Lyu S, Schalk G, et al. Deep feature learning using target priors with applications in ECoG signal decoding for BCI [C] // Proceedings of the Twenty-third International Joint Conference on Artificial Intelligence. Beijing: AAAI Press, 2013, 1785-1791. [22] Vincent P, Larochelle H, Lajoie I, et al. Stacked Denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion [J]. J Mach Learn Res, 2010, 11:3371-3408. [23] Hasperue W. The master algorithm: how the quest for the ultimate learning machine will remake our world [J]. J Comput Sci Tech, 2015, 15(2):157-158. [24] Elman JL. Finding structure in time [J]. Cogn Sci, 1990, 14(2): 179-211. [25] Ruiz L, Cuellar M, Calvo-Flores M. An application of non-linear autoregressive neural networks to predict energy consumption in public buildings [J]. Energies, 2016, 9(9): 684-684. [26] Lang KJ, Waibel A, Hinton GE. A time-delay neural network architecture for isolated word recognition [J]. Neural Networks, 1990, 3(1): 23-43. [27] Maass W, Natschläger T, Markram H. Real-time computing without stable states: A new framework for neural computation based on perturbations[J]. Neural computation, 2002, 14(11): 2531-2560. [28] Li Xiaodong, Ho JKL, Chow TWS. Approximation of dynamical time-variant systems by continuous-time recurrent neural networks [J]. IEEE Trans Circuits Syst II Express Briefs, 2005, 52(10): 656-660. [29] Feldkamp LA, Puskorius GV. A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification [J]. Proc IEEE, 1998, 86(11): 2259-2277. [30] Hüsken M, Stagge P. Recurrent neural networks for time series classification [J]. Neurocomputing, 2003, 50:223-235. [31] Jin L, Nikiforuk PN, Gupta MM. Approximation of discrete-time state-space trajectories using dynamic recurrent neural networks [J]. IEEE Trans Automat Contr, 1995, 40(7): 1266-1270. [32] Schafer AM, Zimmermann H. Recurrent neural networks are universal approximators [J]. Int J Neural Syst, 2007, 17(4): 253-263. [33] Guler NF, Ibeyli ED, Guler I. Recurrent neural networks employing Lyapunov exponents for EEG signals classification [J]. Expert Syst Appl, 2005, 29(3): 506-514. [34] Ren Yuanfang, Wu Yan. Convolutional deep belief networks for feature extraction of EEG signal [C] // 2014 International Joint Conference on Neural Networks (IJCNN). Beijing: IEEE, 2014, 2850-2853. [35] Li Jingcong, Yu Zhu Liang, Gu Zhenghui, et al. A hybrid network for ERP detection and analysis based on restricted boltzmann machine [J]. IEEE Trans Neural Syst Rehab Eng, 2018, 26(3): 563-572. [36] Ma Teng, Li Hui, Yang Hao, et al. The extraction of motion-onset VEP BCI features based on deep learning and compressed sensing [J]. J Neurosci Methods, 2017, 275:80-92. [37] Lu Na, Li Tengfei, Ren Xiaodong, et al. A deep learning scheme for motor imagery classification based on restricted boltzmann machines [J]. IEEE Trans Neural Syst Rehab Eng, 2017, 25(6): 566-576. [38] Ahmed S, Merino LM, Mao Zijing, et al. A deep learning method for classification of images RSVP events with EEG data. [C] //2013 IEEE Global Conference on Signal and Information Processing. Austin:IEEE, 2014: 33-36. [39] Gandhi V, Prasad G, Coyle D, et al. Quantum neural network-based EEG filtering for a brain-computer interface [J]. IEEE Trans Neural Netw Learn Syst, 2014, 25(2): 278-288. [40] Lawhern VJ, Solon AJ, Waytowich NR, et al. EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces[J]. J Neural Eng, 2018, 15(5): 056013. [41] Cecotti H. A time-frequency convolutional neural network for the offline classification of steady-state visual evoked potential responses [J]. Pattern Recognit Lett, 2011, 32(8): 1145-1153. [42] Cecotti H, Graser A. Convolutional neural networks for P300 detection with application to brain-computer interfaces [J]. IEEE Trans Pattern Anal Mach Intell, 2011, 33(3): 433-445. [43] Tang Zhichuan, Li Chao, Sun Shouqian. Single-trial EEG classification of motor imagery using deep convolutional neural networks [J]. Optik, 2017, 130:11-18. [44] Hartmann KG, Schirrmeister RT, Ball T. Hierarchical internal representation of spectral features in deep convolutional networks trained for EEG decoding [C] // International Conference on Brain Computer Interface. South Korea: Gangwon, 2018: 1-6. [45] Nurse E, Mashford BS, Yepes AJ, et al. Decoding EEG and LFP signals using deep learning [C] // Proceedings of the ACM International Conference on Computing Frontiers. Como: ACM, 2016: 259-266. [46] Völker M, Schirrmeister RT, Fiederer LDJ, et al. Deep transfer learning for error decoding from non-invasive EEG [C] //2018 6th International Conference on Brain-Computer Interface (BCI). Gangwoon: IEEE, 2018: 1-6. [47] Abbas W, Khan NA. DeepMI: deep learning for multiclass motor imagery classification [C]//Proceedings of the 2018 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Hawaii:IEEE, 2018: 219-222. [48] Xu Baoguo, Zhang Linlin, Song Aiguo, et al. Wavelet Transform Time-Frequency Image and Convolutional Network-Based Motor Imagery EEG Classification[J]. IEEE Access, 2019, 7: 6084-6093. [49] Peng Weiwei, Hu Li, Zhang Zhiguo, et al. Changes of spontaneous oscillatory activity to tonic heat pain [J]. PLoS ONE, 2014, 9(3):e91052. [50] Liu Wei, Zheng Weilong, Lu Baoliang. Multimodal emotion recognition using multimodal deep learning [J]. arXiv, 2016, 1602:08225. [51] Jirayucharoensak S, PanNgum S, Israsena P. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation [J]. Scientific World Journal, 2014, 2014:627892. [52] Chai Xin, Wang Qisong, Zhao Yongping, et al. Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition [J]. Comput Biol Med, 2016, 79:205-214. [53] Wu EQ, Peng XY, Zhang CZ, et al. Pilots' Fatigue Status Recognition Using Deep Contractive Autoencoder Network[J]. IEEE Trans Instrum Meas, 2019, 1-13. [54] Zheng Weilong, Zhu Jjiayi, Peng Yong, et al. EEG-based emotion classification using deep belief networks [C] //IEEE International Conference on Multimedia and Expo (ICME). Chengdu: IEEE, 2014: 1-6. [55] Chai Rifai, Ling SH, San PP, et al. Improving EEG-based driver fatigue classification using sparse-deep belief networks [J]. Front Neurosci, 2017, 11:103-103. [56] Saha A, Konar A, Chatterjee A, et al. EEG analysis for olfactory perceptual-ability measurement using a recurrent neural classifier [J]. IEEE Trans Hum Mach Syst, 2014, 44(6): 717-730. [57] Bashivan P, Rish I, Yeasin M, et al. Learning representations from EEG with deep recurrent-convolutional neural networks [J]. arXiv,2015,1511:06448. [58] Hajinoroozi M, Mao Zijing, Jung TP, et al. EEG-based prediction of driver's cognitive performance by deep convolutional neural network [J]. Signal Process Image Commun, 2016, 47:549-555. [59] Zeng Hong, Yang Chen, Dai Guojun, et al. EEG classification of driver mental states by deep learning [J]. Cogn Neurodyn. 2018, 12(6):597-606. [60] Lee M, Yeom SK, Baird B, et al. Spatio-temporal analysis of EEG signal during consciousness using convolutional neural network [C]//2018 6th International Conference on Brain-Computer Interface (BCI). Gangwoon: IEEE, 2018:1-3. [61] Han Mei, Xu Xiangmin. EEG-based emotion classification using convolutional neural network [C] //2017 International Conference on Security, Pattern Analysis, and Cybernetics. Shenzhen: IEEE, 2017: 130-135. [62] Tripathi S, Acharya S, Sharma RD, et al. Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset [C] // Proceedings of the 29th AAAI Conference on Artificial Intelligence. San Francisco:AAAI Press, 2017: 4746-4752. [63] Stober S, Cameron DJ, Grahn JA. Using convolutional neural networks to recognize rhythm stimuli from electroencephalography recordings [C] //Neural Information Processing Ssystems. Montreal: MIT Press Cambridge, 2014: 1449-1457. [64] Zhang Pengbo, Wang Xue, Zhang Weihang, et al. Learning Spatial–Spectral–Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment[J]. IEEE Trans Neural Syst Rehab Eng, 2019, 27(1): 31-42. [65] Gao Zhongke, Wang Xinmin, Yang Yuxuan, et al. EEG-based spatio-temporal convolutional neural network for driver fatigue evaluation[J]. IEEE Trans Neural Netw Learn Syst, 2019,1-9. [66] Aljumeily D, Iram S, Vialatte F, et al. A novel method of early diagnosis of Alzheimer’s disease based on EEG signals [J]. Scientific World Journal, 2015, 2015:931387. [67] Samiee K, Kovacs P, Gabbouj M. Epileptic seizure classification of EEG time-series using rational discrete short-time fourier transform [J]. IEEE Trans Biomed Eng, 2015, 62(2): 541-552. [68] Shim M, Hwang HJ, Kim DW, et al. Machine-learning-based diagnosis of schizophrenia using combined sensor-level and source-level EEG features [J]. Schizophr Res, 2016, 176(2-3): 314-319. [69] Qi Yu, Wang Yueming, Zhang Jianmin, et al. Robust deep network with maximum correntropy criterion for seizure detection [J]. Biomed Res Int, 2014, 2014:703816. [70] Turner JT, Page A, Mohsenin T, et al. Deep belief networks used on high resolution multichannel electroencephalography data for seizure detection [J]. arXiv, 2017, 1708:08430. [71] Wulsin D, Gupta JR, Mani R, et al. Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement [J]. J Neural Eng, 2011, 8(3): 036015. [72] Taqi AM, AL-Azzo F, Mariofanna M, et al. Classification and discrimination of focal and non-focal EEG signals based on deep neural network [C] //International Conference on Current Research in Computer Science and Information Technology. Slemani: IEEE, 2017: 86-92.. [73] Hsu YL, Yang YT, Wang JS, et al. Automatic sleep stage recurrent neural classifier using energy features of EEG signals [J]. Neurocomputing, 2013, 104:105-114. [74] Langkvist M, Karlsson L, Loutfi A. Sleep stage classification using unsupervised feature learning [J]. Advances in Artificial Neural Systems, 2012,2012: 107046. [75] Rajaguru H, Prabhakar SK. A Unique Approach to epilepsy classification from EEG signals using dimensionality reduction and neural networks [J]. Circuits Syst, 2016, 7(08): 1455-1464. [76] Ahmedt-Aristizabal D, Fookes C, Nguyen K, et al. Deep classification of epileptic signals [C]//Proceedings of the 2018 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Hawaii:IEEE, 2018: 332-335. [77] Hussein R, Palangi H, Ward RK, et al. Optimized deep neural network architecture for robust detection of epileptic seizures using EEG signals[J]. Clinical Neurophysiology, 2019, 130(1): 25-37. [78] Sun Haoqi, Nagaraj SB, Akeju O, et al. Brain monitoring of sedation in the intensive care unit using a recurrent neural network [C] //Proceedings of the 2018 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Hawaii:IEEE, 2018: 4772-4775. [79] Page A, Shea C, Mohsenin T. Wearable seizure detection using convolutional neural networks with transfer learning [C] //2016 IEEE International Symposium on Circuits and Systems (ISCAS). Montréal: IEEE, 2016: 1086-1089. [80] Acharya UR, Oh SL, Hagiwara Y, et al. Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals [J]. Comput Biol Med, 2018,100:270-278. [81] Thomas J, Comoretto L, Jin Jing, et al. EEG classification via convolutional neural network-based interictal epileptiform event detection [C] //Proceedings of the 2018 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Hawaii:IEEE, 2018: 3148-3151. [82] O’Shea A, Lightbody G, Boylan G, et al. Investigating the impact of CNN depth on neonatal seizure detection performance [C] //Proceedings of the 2018 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Hawaii:IEEE, 2018: 5862-5865. [83] Chanbon S, Thorey V, Arnal PJ, et al. A deep learning architecture to detect events in EEG signals during sleep[J]. arXiv, 2018, 1807:05981. [84] Ieracitano C, Mammone N, Bramanti A, et al. A Convolutional Neural Network approach for classification of dementia stages based on 2D-spectral representation of EEG recordings[J]. Neurocomputing, 2019, 323: 96-107. [85] Li Yitong, Murias M, Major S, et al. Targeting EEG/LFP synchrony with neural nets [C] //The 31st Annual Conference on Neural Information Processing Systems. Long Beach: Curran Associates, 2017: 1-11.
