Abstract：Emotion recognition based on EEG signals plays an important role in the field of human-computer interaction. However, due to the time-varying nature of EEG signals (features may vary significantly at different time periods) and multi-scale characteristics (different features are exhibited at different time and spatial scales), existing deep learning methods often struggle to comprehensively capture and extract various emotion related features from EEG signals. To extract the rich emotional information contained in the time-frequency spatial features of EEG signals, an EEG emotion recognition model that integrates convolutional neural networks (CNN), temporal convolutional networks (TCN), and transformer attention mechanism, frequency spatiotemporal attention temporal convolutional networks (FSA-TCN), was proposed in this work. Firstly, the CNN frequency spatiotemporal convolution layer was used to learn frequency domain information, spatial information, and time domain information, and extracted the frequency spatiotemporal features of EEG signals. Next, the TCN was fused with the Transformer attention mechanism to capture the temporal dependence of frequency spatiotemporal fusion features and extract deep EEG fusion features. Finally, the deep EEG fusion features were input into the fully connected layer for classification. This model conducted ablation experiments and subject dependent and cross subject EEG emotion recognition experiments on 76 800 EEG data samples on the DEAP dataset to verify the effectiveness of each module of the model and its effectiveness in EEG emotion recognition. It achieved emotion recognition accuracy of 92.96% and 92.90% in the valence and arousal dimensions, respectively. In addition, the model's generalization performance was validated on the SEED dataset, and its ability to recognize emotions across datasets was evaluated. The results indicated that the model was able to extract frequency spatiotemporal fusion features of EEG signals and mine deep EEG fusion features and achieved high-precision EEG emotion recognition.
[1] Mauss IB, Robinson MD. Measures of emotion: a reviews [J]. Cognition and Emotion, 2009, 23(2): 209-237. [2] Black MH, Chen NTM, Iyer KK, et al. Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography[J]. Neuroscience & Biobehavioral Reviews, 2017, 80: 488-515. [3] Islam MR, Ahmad M. Virtual image from EEG to recognize appropriate emotion using convolutional neural network[C]//2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT). Dhaka: IEEE, 2019: 1-4. [4] Koelstra S, Muhl C, Soleymani M, et al. Deap: A database for emotion analysis; using physiological signals[J]. IEEE Transactions on Affective Computing, 2011, 3(1): 18-31. [5] Ozdemir MA, Degirmenci M, Izci E, et al. EEG-based emotion recognition with deep convolutional neural networks[J]. Biomedical Engineering/Biomedizinische Technik, 2021, 66(1): 43-57. [6] Hwang S, Hong K, Son G, et al. Learning CNN features from DE features for EEG-based emotion recognition[J]. Pattern Analysis and Applications, 2020, 23: 1323-1335. [7] Duan Ruonan, Zhu Jiayi, Lu BaoLiang. Differential entropy feature for EEG-based emotion classification[C]//2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). San Diego: IEEE, 2013: 81-84. [8] He J, Zhao L, Yang H, et al. HSI-BERT: Hyperspectral image classification using the bidirectional encoder representation from transformers [J]. IEEE Transactions on Geoscience and Remote Sensing,2020,58(1):165-178. [9] Lea C, Flynn MD, Vidal R, et al. Temporal convolutional networks for action segmentation and detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE,2017: 156-165. [10] Yang Liuqing, Liu Jiwei. EEG-based emotion recognition using temporal convolutional network[C]//2019 IEEE 8th Data Driven Control and Learning Systems Conference (DDCLS). Dali:IEEE, 2019: 437-442. [11] Samavat A, Khalili E, Ayati B, et al. Deep learning model with adaptive regularization for EEG-based emotion recognition using temporal and frequency features[J]. IEEE Access, 2022, 10: 24520-24527. [12] Kim Y, Choi A. EEG-based emotion classification using long short-term memory network with attention mechanism[J]. Sensors, 2020, 20(23): 6727. [13] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, New York: Curran Associates Inc,2017:5998-6008. [14] He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas:IEEE,2016: 770-778. [15] Ding Yi, Zhang Sun, Tang Chuangao, et al. MASA-TCN: multi-anchor space-aware temporal convolutional neural networks for continuous and discrete EEG emotion recognition[J]. IEEE Journal of Biomedical and Health Informatics, 2024,28(7):3953-3964. [16] Ding Yi, Robinson N, Zhang Sun, et al. Tsception: capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition[J]. IEEE Transactions on Affective Computing, 2022, 14(3): 2238-2250. [17] Bello I, Zoph B, Vaswani A, et al. Attention augmented convolutional networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3286-3295. [18] Kinga D, Adam JB. Adam: A method for stochastic optimization[C]//International Conference on Learning Representations (ICLR). San Diego: CORR, 2015, 5: 6. [19] Yang Yilong, Wu Qingfeng, Qiu Ming, et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]//2018 International Joint Conference on Neural Networks (IJCNN). Rio de Janeiro: IEEE, 2018: 1-7. [20] Tao Wei, Li Chang, Song Rencheng, et al. EEG-based emotion recognition via channel-wise attention and self attention [J]. IEEE Transactions on Affective Computing, 2020, 14(1): 382-393. [21] Yao Qunli, Gu Heng, Wang Shaodi, et al. A feature-fused convolutional neural network for emotion recognition from multichannel EEG signals[J]. IEEE Sensors Journal, 2022, 22(12): 11954-11964. [22] Alhagry S, Fahmy AA, El-Khoribi RA. Emotion recognition based on EEG using LSTM recurrent neural network[J]. International Journal of Advanced Computer Science and Applications, 2017, 8(10):358. [23] An Yi, Xu Ning, Qu Zhen. Leveraging spatial-temporal convolutional features for EEG-based emotion recognition[J]. Biomedical Signal Processing and Control, 2021, 69: 102743. [24] Salama ES, EL-Khoribi RA, Shoman ME, et al. EEG-based emotion recognition using 3D convolutional neural networks[J]. International Journal of Advanced Computer Science and Applications, 2018, 9(8):337. [25] Zhu Yongsheng, Zhong Qinghua. Differential entropy feature signal extraction based on activation mode and its recognition in convolutional gated recurrent unit network[J]. Frontiers in Physics, 2021, 8: 629620. [26] Zhang Xiaodan, Li Yige, Du Jinxiang, et al. Feature pyramid networks and long short-term memory for EEG feature map-based emotion recognition[J]. Sensors, 2023, 23(3): 1622.
