Abstract：In recent years, the research of emotion recognition based on EEG signals has gradually made remarkable progress. However, the labeling of labels requires a lot of manpower, and it is difficult to quickly obtain a large number of labeled data in practical applications. Therefore, how to utilize limited labels efficiently in the emotion recognition research become one bottleneck problem to overcome. In this work, a model architecture based on self-supervised double-flow twin network was proposed, which consisted of two interacting and learning branches of convolutional neural networks. First, the model was pre-trained. The amplified data of the input signal after two random signal transformations were input into the training branch and the target branch of the twin network. After extracting features from the convolutional module and the fully connected module in the branch, the model learned the general representation of the EEG signal in the process. Finally, the encoder part of the training branch was retained, and then the fully connected layer was used to fine-tune the model with labeled data, and the classification results are obtained. Data samples from public data sets SEED and SEED-IV were used to verify and evaluate the model classification effect. Under the fully labeled data, the classification accuracy of 93.92% and 89.71% were achieved, respectively. Under 50% label usage, the average accuracy of the three categories was 92.68%, which was only 1.24% less than that using all labels. The results showed that the model effectively extracted the general representation of EEG data, and achieved high recognition accuracy with relatively less labels.
[1] Picard RW, Vyzas E, Healey J. Toward machine emotional intelligence: Analysis of affective physiological state[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(10): 1175-1191.
[2] Kleinsmith A, Bianchi-Berthouze N. Affective body expression perception and recognition: a survey[J]. IEEE Transactions on Affective Computing, 2012, 4(1): 15-33.
[3] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE/CVF, 2016: 770-778.
[4] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE/CVF, 2015: 1-9.
[5] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.
[6] Emsawas T, Morita T, Kimura T, et al. Multi-kernel temporal and spatial convolution for EEG-based emotion classification[J]. Sensors, 2022, 22(21): 8250.
[7] Gao Z, Wang X, Yang Y, et al. A channel-fused dense convolutional network for EEG-based emotion recognition[J]. IEEE Transactions on Cognitive and Developmental Systems, 2020, 13(4): 945-954.
[8] Du X, Ma C, Zhang G, et al. An efficient LSTM network for emotion recognition from multichannel EEG signals[J]. IEEE Transactions on Affective Computing, 2020, 13(3): 1528-1540.
[9] 赵红宇,李畅,刘羽,等. 基于无源域适应的脑电情绪识别[J]. 中国生物医学工程学报, 2024, 43(2): 129-142.
[10] Yin J, Wu M, Yang Y, et al. Research on multimodal emotion recognition based on fusion of electroencephalogram and electrooculography[J]. IEEE Transactions on Instrumentation and Measurement, 2024, 73: 1-12.
[11] Wang Y, Song W, Tao W, et al. A systematic review on affective computing: emotion models, databases, and recent advances[J]. Information Fusion, 2022, 83: 19-52.
[12] Yang Z, Li D, Hou F, et al. Deep feature extraction and attention fusion for multimodal emotion recognition[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2023, 71(3): 1526-1530.
[13] Chang S, Yang Z, You Y, et al. DSSNet: a deep sequential sleep network for self-supervised representation learning based on single-channel EEG[J]. IEEE Signal Processing Letters, 2022, 29: 2143-2147.
[14] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE/CVF, 2020: 9729-9738.
[15] Banville H, Chehab O, Hyvärinen A, et al. Uncovering the structure of clinical EEG signals with self-supervised learning[J]. Journal of Neural Engineering, 2021, 18(4): 046020.
[16] Jiang X, Zhao J, Du B, et al. Self-supervised contrastive learning for EEG-based sleep staging[C]//2021 International Joint Conference on Neural Networks (IJCNN). Prague: IEEE, 2021: 1-8.
[17] Mohsenvand MN, Izadi MR, Maes P. Contrastive representation learning for electroencephalogram classification[C]//Machine Learning for Health. Virtual:PMLR, 2020: 238-253.
[18] Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[C]//International Conference on Machine Learning. Vienna :PMLR, 2020: 1597-1607.
[19] Wu D, Li S, Yang J, et al. neuro2vec: Masked fourier spectrum prediction for neurophysiological representation learning[J/OL]. arXiv preprint arXiv: 2204. 12440v1, 2022-04-20/2024-02-01.
[20] Zheng WL, Lu BL. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2015, 7(3): 162-175.
[21] Duan RN, Zhu JY, Lu BL. Differential entropy feature for EEG-based emotion classification[C]//2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). San Diego: IEEE, 2013: 81-84.
[22] Zheng W L, Liu W, Lu Y, et al. Emotionmeter: a multimodal framework for recognizing human emotions[J]. IEEE Transactions on Cybernetics, 2018, 49(3): 1110-1122.
[23] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J/OL]. arXiv preprint arXiv: 1503. 02531v1, 2015-03-09/2024-02-01.
[24] Bachman P, Hjelm RD, Buchwalter W. Learning representations by maximizing mutual information across views[J]. Advances in Neural Information Processing Systems, 2019, 32: 15509-15519.
[25] Loshchilov I, Hutter F. Sgdr: stochastic gradient descent with warm restarts[J/OL]. arXiv preprint arXiv: 1608. 03983v1, 2016-05-03/2024-02-01.
[26] Li T H, Liu W, Zheng W L, et al. Classification of five emotions from EEG and eye movement signals: discrimination ability and stability over time[C]//2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). San Francisco: IEEE, 2019: 607-610.
[27] Zhong P, Wang D, Miao C. EEG-based emotion recognition using regularized graph neural networks[J]. IEEE Transactions on Affective Computing, 2020, 13(3): 1290-1301.
[28] Zhang T, Zheng W, Cui Z, et al. Spatial-temporal recurrent neural network for emotion recognition[J]. IEEE Transactions on Cybernetics, 2018, 49(3): 839-847.
[29] Moon SE, Jang S, Lee JS. Convolutional neural network approach for EEG-based emotion recognition using brain connectivity and its spatial information[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Calgary : IEEE, 2018: 2556-2560.
[30] Li Y, Zheng W, Cui Z, et al. A novel neural network model based on cerebral hemispheric asymmetry for EEG emotion recognition[C]//IJCAI. Stockholm: Curran Associates, Inc.,2018: 1561-1567.
[31] Li Y, Wang L, Zheng W, et al. A novel bi-hemispheric discrepancy model for EEG emotion recognition[J]. IEEE Transactions on Cognitive and Developmental Systems, 2020, 13(2): 354-367.
[32] Li Y, Zheng W, Wang L, et al. From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition[J]. IEEE Transactions on Affective Computing, 2019, 13(2): 568-578.
[33] Cheah KH, Nisar H, Yap VV, et al. Optimizing residual networks and vgg for classification of eeg signals: Identifying ideal channels for emotion recognition[J]. Journal of Healthcare Engineering, 2021, 2021(1): 5599615.
[34] Miljevic A, Bailey NW, Murphy OW, et al. Alterations in EEG functional connectivity in individuals with depression: a systematic review[J]. Journal of Affective Disorders, 2023, 328: 287-302.
