Abstract：Electroencephalography (EEG) has millisecond time resolution, which can acquire real-time neurophysiology information of brain activity. EEG has been widely used in cerebral imaging and has become an important tool in neuroscience and neuroengineering in 21st century. However, the original signal-to-noise ratio (SNR) and spatial resolution are poor and the decoding is seriously hindered due to the volume condution effect. With the great development of Deep Learning (DL) in this century, researchers have been trying to combine the two to explore the application of deep learning in EEG data processing, and leading to some phase achievements. Nevertheless, there are still challenges in applying the current DNNs to EEG data processing. Combining with recent studies on DNN-based EEG data processing, this paper introduced the implementation of deep EEG decoding and discussed existing problems and future directions.
刘政, 何峰, 汤佳贝, 万柏坤, 明东. 基于深度学习的头皮脑电信息解码研究进展[J]. 中国生物医学工程学报, 2020, 39(2): 215-228.
Liu Zheng, He Feng, Tang Jiabei, Wan Baikun, Ming Dong. Research Advancements of Deep Learning on EEG Decoding. Chinese Journal of Biomedical Engineering, 2020, 39(2): 215-228.
[1] 李恩中, 高家红, 卢光明,等. 神经功能成像及其在重大脑疾病中的应用 [J]. 中国科学:生命科学, 2015, 45(3): 237-246.
[2] Hinton GE, Osindero S,Teh Y-W. A fast learning algorithm for deep belief nets [J]. Neural computation, 2006, 18(7): 1527-1554.
[3] He Kaiming, Zhang Xiangyu, Ren Shaoqing,et al. Deep residual learning for image recognition [C] //Proceedings of the IEEE conference on computer vision and pattern recognition. New York: IEEE, 2016: 770-778.
[4] Lotte F, Bougrain L, Cichocki A,et al. A review of classification algorithms for EEG-based brain-computer interfaces: A 10 year update [J]. Journal of neural engineering, 2018, 15(3): 031005.
[5] Cai Hanshu, Sha Xiaocong, Han Xue,et al. Pervasive EEG diagnosis of depression using deep belief network with three-electrodes EEG collector [C] //2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). New York: IEEE, 2016: 1239-1246.
[6] Jiao Zhicheng, Gao Xinbo, Wang Ying,et al. Deep convolutional neural networks for mental load classification based on EEG data [J]. Pattern Recognition, 2018, 76: 582-595.
[7] Chai R, Ling SH, San PP,et al. Improving EEG-based driver fatigue classification using sparse-deep belief networks [J]. Frontiers in Neuroscience, 2017, 11: 103.
[8] 单绍杰, 李汉军, 王璐璐,等. 基于LSTM模型的单导联脑电癫痫发作预测 [J]. 计算机应用研究, 2018, 35(11): 3251-3254.
[9] Zheng Weilong,Lu Baoliang. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks [J]. IEEE Transactions on Autonomous Mental Development, 2015, 7(3): 162-175.
[10] Yang Huijuan, Sakhavi S, Ang KK,et al. On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of EEG signals classification [C] //2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2015: 2620-2623.
[11] Yu Zhongliang, Song Jinchun. Multi-class motor imagery classification by singular value decomposition and deep boltzmann machine [C] //2017 IEEE 3rd Information Technology and Mechatronics Engineering Conference (ITOEC). New York: IEEE, 2017: 376-379.
[12] LeCun Y, Boser B, Denker JS,et al. Backpropagation applied to handwritten zip code recognition [J]. Neural computation, 1989, 1(4): 541-551.
[13] Krizhevsky A, Sutskever I,Hinton GE. Imagenet classification with deep convolutional neural networks [C] //Fernando P, Christopher JCB, Léon B,Kilian QW.Advances in neural information processing systems 25. (NIPS 2012). New York: Curran Associates, 2012: 1097-1105.
[14] Zeiler MD,Fergus R. Visualizing and understanding convolutional networks [C] // European Conference on Computer Vision. Berlin: Springer, 2014: 818-833.
[15] LeCun Y, Bengio Y. Convolutional networks for images, speech, and time series [J]. The handbook of Brain Theory and Neural Networks, 1995, 3361(10): 1995-2000.
[16] Schirrmeister RT, Springenberg JT, Fiederer LDJ,et al. Deep learning with convolutional neural networks for EEG decoding and visualization [J]. Human Brain Mapping, 2017, 38(11): 5391-5420.
[17] Goh SK, Abbass HA, Tan KC,et al. Spatio-spectral representation learning for electroencephalographic gait-pattern classification [J]. IEEE Transactions on Neural Systems Rehabilitation Engineering, 2018, 26(9): 1858-1867.
[18] Qiao Rui, Qing Chunmei, Zhang Tong,et al. A novel deep-learning based framework for multi-subject emotion recognition [C] //2017 4th International Conference on Information, Cybernetics and Computational Social Systems (ICCSS). New York: IEEE, 2017: 181-185.
[19] Xu Baoguo, Zhang Linlin, Song Aiguo,et al. Wavelet transform time-frequency image and convolutional network-based motor imagery EEG classification [J]. IEEE Access, 2019, 7: 6084-6093.
[20] Ma Teng, Li Hui, Yang Hao,et al. The extraction of motion-onset VEP BCI features based on deep learning and compressed sensing [J]. Journal of Neuroscience Methods, 2017, 275: 80-92.
[21] Tabar YR,Halici U. A novel deep learning approach for classification of EEG motor imagery signals [J]. Journal of Neural Engineering, 2016. 14(1): 016003.
[22] Nurse E, Mashford BS, Yepes AJ,et al. Decoding EEG and LFP signals using deep learning: heading TrueNorth [C] //Proceedings of the ACM International Conference on Computing Frontiers. New York: ACM, 2016: 259-266.
[23] Pereira A, Padden D, Jantz J,et al. Cross-subject EEG event-related potential classification for brain-computer interfaces using residual networks [DB/OL]. http://hal.archives-ouvertes.fr/hal-01878227. 2018-09-20/2019-04-29.
[24] 孔祥浩, 马琳, 薄洪健,等. CNN与CSP相结合的脑电特征提取与识别方法研究 [J]. 信号处理, 2018, 34(2): 164-173.
[25] Sakhavi S, Guan C,Yan S. Learning temporal information for brain-computer interface using convolutional neural networks [J]. IEEE Transactions on Neural Networks Learning Systems, 2018, 29(11): 5619-5629.
[26] Hajinoroozi M, Zhang JM,Huang Yufei. Driver's fatigue prediction by deep covariance learning from EEG [C] //2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). New York: IEEE, 2017: 240-245.
[27] 李海峰, 徐聪, 马琳. 基于C-LSTM模型的端到端多粒度运动想象脑电信号分析方法 [J]. 信号处理, 2018, 34(8): 883-890.
[28] Hajinoroozi M, Mao Zijing,Huang Yufei. Prediction of driver′s drowsy and alert states from EEG signals with deep learning [C] //2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP). New York: IEEE, 2015: 493-496.
[29] Kiranyaz S, Ince T,Gabbouj M. Real-time patient-specific ECG classification by 1-D convolutional neural networks [J]. IEEE Transactions on Biomedical Engineering, 2016, 63(3): 664-675.
[30] Aznan NKN, Bonner S, Connolly J,et al. On the classification of SSVEP-based dry-EEG signals via convolutional neural networks [C] //2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). New York: IEEE, 2018: 3726-3731.
[31] Cecotti H,Graser A. Convolutional neural networks for P300 detection with application to brain-computer interfaces [J]. IEEE Transactions on Pattern Analysis Machine Intelligence, 2011, 33(3): 433-445.
[32] Dose H, Mller JS, Puthusserypady S,et al. A deep learning MI-EEG classification model for BCIs [C] //2018 26th European Signal Processing Conference (EUSIPCO). New York: IEEE, 2018: 1676-1679.
[33] Van Leeuwen K, Sun H, Tabaeizadeh M,et al. Detecting abnormal electroencephalograms using deep convolutional networks [J]. Clinical Neurophysiology, 2019, 130(1): 77-84.
[34] Tang Zhichuan, Li Chao, Sun Shouqian. Single-trial EEG classification of motor imagery using deep convolutional neural networks [J]. Optik-International Journal for Light Electron Optics, 2017, 130: 11-18.
[35] Sakkalis V. Review of advanced techniques for the estimation of brain connectivity measured with EEG/MEG [J]. Computers in Biology Medicine, 2011, 41(12): 1110-1117.
[36] Moon SE, Jang S,Lee JS. Convolutional neural network approach for EEG-based emotion recognition using brain connectivity and its spatial information [C] //2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New York: IEEE, 2018: 2556-2560.
[37] Li Youjun, Huang Jiajin, Zhou Haiyan,et al. Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks [J]. Applied Sciences, 2017, 7(10): 1060-1080.
[38] Bashivan P, Rish I, Yeasin M,et al. Learning representations from EEG with deep recurrent-convolutional neural networks [DB/OL]. http://arxiv.org/abs/1511.06448. 2016-02-29/2019-04-29.
[39] Salama ES, El-Khoribi RA, Shoman ME,et al. EEG-Based Emotion Recognition using 3D Convolutional Neural Networks [J]. International Journal of Advanced Computer Science Applications, 2018, 9(8): 329-337.
[40] Wei Xiaoyan, Zhou Lin, Chen Ziyi,et al. Automatic seizure detection using three-dimensional CNN based on multi-channel EEG [J]. BMC Medical Informatics Decision Making, 2018, 18(5): 111.
[41] 杨丽, 吴雨茜, 王俊丽,等. 循环神经网络研究综述 [J]. 计算机应用, 2018, 38(S2): 1-6,26.
[42] Werbos P. Backpropagation through time: what it does and how to do it [J]. Proceedings of the IEEE, 1990, 78(10): 1550-1560.
[43] Bengio Y, Simard P,Frasconi P. Learning long-term dependencies with gradient descent is difficult [J]. IEEE Transactions on Neural Networks, 1994, 5(2): 157-166.
[44] Hochreiter S,Schmidhuber J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780.
[45] Ahmedt-Aristizabal D, Fookes C, Nguyen K,et al. Deep classification of epileptic signals [C] //2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2018: 332-335.
[46] Ma Xuelin, Qiu Shuang, Du Changde,et al. Improving EEG-based motor imagery classification via spatial and temporal recurrent neural networks [C] //2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2018: 1903-1906.
[47] Chen Weitong, Wang Sen, Zhang Xiang,et al. EEG-based motion intention recognition via multi-task RNNs [C] // Proceedings of the 2018 SIAM International Conference on Data Mining. Philadelphia: SIAM, 2018: 279-287.
[48] Soleymani M, Asghari-Esfeden S, Fu Yun,et al. Analysis of EEG signals and facial expressions for continuous emotion detection [J]. IEEE Transactions on Affective Computing, 2016, 7(1): 17-28.
[49] Hussein R, Palangi H, Ward RK,et al. Optimized deep neural network architecture for robust detection of epileptic seizures using EEG signals [J]. Clinical Neurophysiology, 2019, 130(1): 25-37.
[50] Ni Zhaoheng, Yuksel AC, Ni Xiuyan,et al. Confused or not confused?: Disentangling brain activity from eeg data using bidirectional lstm recurrent neural networks [C] //Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics. New York: ACM, 2017: 241-246.
[51] Thodoroff P, Pineau J,Lim A. Learning robust features using deep learning for automatic seizure detection [C] //Machine Learning for Healthcare Conference. Cambridge: MIT Press, 2016: 178-190.
[52] Graves A,Schmidhuber J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures [J]. Neural Networks, 2005, 18(5-6): 602-610.
[53] Schuster M,Paliwal KK. Bidirectional recurrent neural networks [J]. IEEE Transactions on Signal Processing, 1997, 45(11): 2673-2681.
[54] Supratak A, Dong Hao, Wu Chao,et al. DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG [J]. IEEE Transactions on Neural Systems Rehabilitation Engineering, 2017, 25(11): 1998-2008.
[55] Li Mingai, Zhu Wei, Zhang Meng,et al. The novel recognition method with optimal wavelet packet and LSTM based recurrent neural network [C] //2017 IEEE International Conference on Mechatronics and Automation (ICMA). New York: IEEE, 2017: 584-589.
[56] Li Xiang, Song Dawei, Zhang Peng,et al. Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network [C] //2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). New York: IEEE, 2016: 352-359.
[57] Tsiouris KM, Pezoulas VC, Zervakis M,et al. A long short-term memory deep learning network for the prediction of epileptic seizures using EEG signals [J]. Computers in Biology Medicine, 2018, 99: 24-37.
[58] Roy S, Kiral-Kornek I,Harrer S. Deep Learning enabled automatic abnormal EEG identification [C] //2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2018: 2756-2759.
[59] Kuanar S, Athitsos V, Pradhan N,et al. Cognitive analysis of working memory load from EEG, by a deep recurrent neural network [C] //2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New York: IEEE, 2018: 2576-2580.
[60] Bresch E, Grossekathofer U,Garcia-Molina G. Recurrent deep neural networks for real-time sleep stage classification from single channel EEG [J]. Frontiers in Computational Neuroscience, 2018, 12: 85.
[61] Zhang Xiang, Yao Lina, Sheng Quan,et al. Converting your thoughts to texts: Enabling brain typing via deep feature learning of EEG signals [C] //2018 IEEE International Conference on Pervasive Computing and Communications. New York: IEEE, 2018: 1-10.
[62] Hinton GE. Learning multiple layers of representation [J]. Trends in Cognitive Sciences, 2007, 11(10): 428-434.
[63] 唐贤伦, 周家林, 张娜,等. 基于深度信念网络的运动想象脑电信号识别 [J]. 信息与控制, 2015, 44(6): 717-721,738.
[64] An Xiu, Kuang Deping, Guo Xiaojiao,et al. A deep learning method for classification of EEG data based on motor imagery [C] //International Conference on Intelligent Computing. Berlin: Springer, 2014: 203-210.
[65] Ren Yuanfang,Wu Yan. Convolutional deep belief networks for feature extraction of EEG signal [C] //2014 International Joint Conference on Neural Networks (IJCNN). New York: IEEE, 2014: 2850-2853.
[66] 张毅, 陈永强,蔡军. 基于DBN结合SVM的脑电信号识别研究 [J]. 控制工程, 2018, 25(06): 1007-1011.
[67] Yin Zhong,Zhang Jianhua. Cross-subject recognition of operator functional states via EEG and switching deep belief networks with adaptive weights [J]. Neurocomputing, 2017, 260: 349-366.
[68] Hajinoroozi M, Jung TP, Lin CT,et al. Feature extraction with deep belief networks for driver's cognitive states prediction from EEG data [C] //2015 IEEE China Summit and International Conference on Signal and Information Processing. New York: IEEE, 2015: 812-815.
[69] Wulsin D, Gupta J, Mani R,et al. Modeling electroencephalography waveforms with semi-supervised deep belief nets: Fast classification and anomaly measurement [J]. Journal of Neural Engineering, 2011, 8(3): 036015.
[70] Lu Na, Li Tengfei, Ren Xiaodong,et al. A deep learning scheme for motor imagery classification based on restricted boltzmann machines [J]. IEEE Transactions on Neural Systems Rehabilitation Engineering, 2017, 25(6): 566-576.
[71] Zheng Weilong, Guo Haotian, Lu Baoliang. Revealing critical channels and frequency bands for emotion recognition from EEG with deep belief network [C] //2015 7th International IEEE/EMBS Conference on Neural Engineering (NER). New York: IEEE, 2015: 154-157.
[72] Goodfellow I, Pouget-Abadie J, Mirza M,et al. Generative adversarial nets [C] //Zoubin G, Max W, Corinna C, Neil DL,Kilian QW.Advances in Neural Information Processing Systems 27 (NIPS 2014). New York: Curran Associates, 2014: 2672-2680.
[73] Abdelfattah SM, Abdelrahman GM,Wang Min. Augmenting the size of EEG datasets using generative adversarial networks [C] //2018 International Joint Conference on Neural Networks (IJCNN). New York: IEEE, 2018: 1-6.
[74] Luo Yun. EEG data augmentation for emotion recognition using a conditional Wasserstein GAN [C] //2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2018: 2535-2538.
[75] Corley IA,Huang Yufei. Deep EEG super-resolution: Upsampling EEG spatial resolution with generative adversarial networks [C] //2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). New York: IEEE, 2018: 100-103.
[76] Horikawa T,Kamitani Y. Generic decoding of seen and imagined objects using hierarchical visual features [J]. Nature Communications, 2017. 8: 15037.
[77] Spampinato C, Palazzo S, Kavasidis I,et al. Deep learning human mind for automated visual classification [C] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 6809-6817.
[78] Palazzo S, Spampinato C, Kavasidis I,et al. Generative adversarial networks conditioned by brain signals [C] //Proceedings of the IEEE International Conference on Computer Vision. New York: IEEE, 2017: 3410-3418.
[79] Kavasidis I, Palazzo S, Spampinato C,et al. Brain2image: Converting brain signals into images [C] //Proceedings of the 25th ACM International Conference on Multimedia. New York: ACM, 2017: 1809-1817.
[80] Tirupattur P, Rawat YS, Spampinato C,et al. ThoughtViz: Visualizing human thoughts using generative adversarial network [C] //ACM Multimedia Conference. New York: ACM, 2018: 950-958.
[81] Lee Y,Huang Yufei. Generating target/non-target images of an RSVP experiment from brain signals in by conditional generative adversarial network [C] //2018 IEEE EMBS international conference on Biomedical & Health Informatics (BHI). New York: IEEE, 2018: 182-185.
[82] 张力新, 张裕坤, 柯余峰,等. 基于Hololens的增强现实脑-机接口研究 [J]. 中国生物医学工程学报, 2019, 38(1): 51-58.
[83] Nakanishi M, Wang Yijun, Chen Xiaogang,et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis [J]. IEEE Transactions on Biomedical Engineering, 2018. 65(1): 104-112.
[84] Waytowich N, Lawhern VJ, Garcia JO,et al. Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials [J]. Journal of Neural Engineering, 2018. 15(6): 066031.
[85] Lawhern VJ, Solon AJ, Waytowich NR,et al. EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces [J]. Journal of Neural Engineering, 2018. 15(5): 056013.
[86] Dose H, Mller JS, Iversen HK,et al. An end-to-end deep learning approach to MI-EEG signal classification for BCIs [J]. Expert Systems with Applications, 2018. 114: 532-542.
[87] Kulasingham J, Vibujithan V,De Silva A. Deep belief networks and stacked autoencoders for the P300 Guilty Knowledge Test [C] //2016 IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES). New York: IEEE, 2016: 127-132.
[88] O'Shea A, Lightbody G, Boylan G,et al. Neonatal seizure detection using convolutional neural networks [C] //2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP). New York: IEEE, 2017: 1-6.
[89] Ullah I, Hussain M,Aboalsamh H. An automated system for epilepsy detection using EEG brain signals based on deep learning approach [J]. Expert Systems with Applications, 2018, 107: 61-71.
[90] Zhao Yilu,He Lianghua. Deep learning in the EEG diagnosis of Alzheimer’s disease [C] //Asian Conference on Computer Vision. Berlin: Springer, 2014: 340-353.
[91] Acharya UR, Oh SL, Hagiwara Y,et al. Automated EEG-based screening of depression using deep convolutional neural network [J]. Computer Methods Programs in Biomedicine, 2018, 161: 103-113.
[92] Zheng Weilong, Zhu Jiayi, Peng Yong, et al. EEG-based emotion classification using deep belief networks [C] //2014 IEEE International Conference on Multimedia and Expo (ICME). New York: IEEE, 2014: 1-6.
[93] Hajinoroozi M, Mao Zijing, Jung TP,et al. EEG-based prediction of driver's cognitive performance by deep convolutional neural network [J]. Signal Processing: Image Communication, 2016, 47: 549-555.
[94] Hefron R, Borghetti B, Schubert Kabban C,et al. Cross-participant EEG-based assessment of cognitive workload using multi-path convolutional recurrent neural networks [J]. Sensors, 2018, 18(5): 1339-1366.
[95] Huang Jungming, Xu Xiangmin,Zhang Tong. Emotion classification using deep neural networks and emotional patches [C] //2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). New York: IEEE, 2017: 958-962.
[96] 汪露雲, 孔万增, 张昕昱,等. 脑纹识别研究综述 [J]. 中国生物医学工程学报, 2017,36(5): 602-607.
[97] Ma Lan, Minett JW, Blu T,et al. Resting state EEG-based biometrics for individual identification using convolutional neural networks [C] //2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2015: 2848-2851.
[98] Das R, Maiorana E,Campisi P. Visually evoked potential for EEG biometrics using convolutional neural network [C] //2017 25th European Signal Processing Conference (EUSIPCO). New York: IEEE, 2017: 951-955.
[99] Gandhi S, Oates T, Mohsenin T,et al. Denoising Time Series Data Using Asymmetric Generative Adversarial Networks [C] //Pacific-Asia Conference on Knowledge Discovery and Data Mining. Berlin: Springer, 2018: 285-296.
