Abstract：The brain topographic map can be used to monitor the state of brain activity. In order to accurately extract the spatial characteristics of the signals generated by the brain activity of the subjects and effectively improve the classification accuracy, a multi-modal brain topographic map neural network classification algorithm (MBTMNN) was proposed by combining the brain topographic map and convolutional neural network to classify and recognize motor imagery and mental arithmetic. Firstly, EEG and fNIRS signals were preprocessed to extract the energy characteristics of EEG and the concentration characteristics of oxygenated hemoglobin in fNIRS. The colormap of all samples was unified by combining with the position of each electrode to generate brain topographic maps. The EEG and fNIRS signals were simultaneously input into the convolutional neural network and fused in the feature layer to obtain a training model. The six-fold cross validation experiment was conducted on the 2017 Berlin EEG/fNIRS public dataset. The dataset included 29 subjects, with 300 samples each, in the four classification scenarios of left versus right hand motor imagery, mental arithmetic versus resting state, motor imagery versus mental arithmetic versus resting state, and left versus right hand motor imagery versus mental arithmetic versus resting state, the accuracy rates were 82.91%, 94%, 90.34% and 78.18%, respectively, which were higher than those of recently reported with the same dataset and the state of single mode method. The results indicated that the proposed method effectively fused EEG and fNIRS signals to improve the classification accuracy.
[1] 陈小刚, 杨晨, 陈菁菁, 等. 脑机接口技术发展新趋势——基于2019-2020年研究进展 [J]. 科技导报, 2021, 39(19): 56-65. [2] 徐若豪, 李太豪, 裴冠雄, 等. 脑机接口技术赋能元宇宙 [J]. 张江科技评论, 2022, (2): 29-31. [3] Fazli S, Mehnert J, Steinbrink J, et al. Enhanced performance by a hybrid NIRS-EEG brain computer interface [J]. NeuroImage, 2012, 59(1): 519-529. [4] Naseer N, Hong KS. Classification of functional near-infrared spectroscopy signals corresponding to the right- and left-wrist motor imagery for development of a brain-computer interface [J]. Neuroscience Letters, 2013, 553: 84-89. [5] Hong KS, Naseer N, Kim YH. Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI [J]. Neuroscience Letters, 2015, 587: 87-92. [6] Jiang X, Gu X, Xu K, et al. Independent decision path fusion for bimodal asynchronous brain-computer interface to discriminate multiclass mental states [J]. IEEE Access, 2019, 7: 165303-165317. [7] Shin J, AV Lühmann, Blankertz B, et al. Open access dataset for EEG+NIRS single-trial classification [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(10): 1735-1745. [8] Ghonchi H, Fateh M, Abolghasemi V, et al. Deep recurrent-onvolutional neural network for classification of simultaneous EEG-NIRS signals [J]. IET Signal Processing, 2020, 14(3): 142-153. [9] 周伊婕, 宋西姊, 何峰, 等. 基于脑电的多模态神经功能成像新技术研究进展 [J]. 中国生物医学工程学报, 2020, 39(5): 595-602. [10] Ergun E, Aydemir O. Classification of motor imaginary based Near- Infrared spectroscopy signals [C] //2018 26th Signal Processing and Communications Applications Conference (SIU). Izmir: IEEE, 2018: 1-4. [11] Rahman MA, Uddin MS, Ahmad M. Modeling and classification of voluntary and imagery movements for brain-computer interface from fNIR and EEG signals through convolutional neural network [J]. Health Information Science and Systems, 2019, 7(1): 22. [12] 周如双, 赵慧琳, 林玮玥, 等. 基于深浅特征融合的深度卷积残差网络的脑电情绪识别模型 [J]. 中国生物医学工程学报, 2021, 40(6): 641-652. [13] Xu M, Yao J, Zhang Z, et al. Learning EEG topographical representation for classification via convolutional neural network [J]. Pattern Recognition, 2020, 105(4): 107390. [14] Ergun E, Aydemir O. Decoding of binary mental arithmetic based near-infrared spectroscopy signals [C] //2018 3rd International Conference on Computer Science and Engineering (UBMK). Bosnia and Herzegovina: IEEE, 2018: 201-204. [15] Vigário RN. Extraction of ocular artefacts from EEG using independent component analysis[J]. Electroencephalography & Clinical Neurophysiology, 1997, 103(3): 395-404. [16] Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis [J]. Journal of Neuroscience Methods, 2004, 134(1): 9-21. [17] Kocsis L, Herman P, Eke A. The modified Beer-Lambert law revisited [J]. Physics in Medicine & Biology, 2006, 51(5): N91. [18] Aydin EA. Subject-Specific feature selection for near infrared spectroscopy based brain-computer interfaces-ScienceDirect [J]. Computer Methods and Programs in Biomedicine, 2020, 195: 105535. [19] Laurens VDM, Hinton G. Visualizing data using t-SNE [J]. Journal of Machine Learning Research, 2008, 9(2605): 2579-2605. [20] Eyndhoven SV, Bousse M, Hunyadi B, et al. Single-channel EEG classification by multi-channel tensor subspace learning and regression [C] //2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). Aalborg: IEEE, 2018: 1-6.
