Abstract：Brain-computer interface (BCI) is one of the most active research directions in the field of neural engineering, which can establish a communication and control pathway between the brain and the external environment independent of peripheral nerves or muscles and is helpful to restore the self-care ability of people with movement disorders. Steady-state visual evoked potential BCI attracts great attention for the high information transfer rate and less requirements of training. The existing non-invasive high speed BCI is mainly from or based on SSVEP. In recent years, multi-modal BCI integrateing SSVEP and other input signals has become a new trend in BCI research for further improvement of BCI performance. This paper reviewed advances of multi-modal BCI combined with SSVEP from several aspects, including the type of input signals, fusion of experimental paradigm and signal, aiming to help readers understand the research trends in this field and inspire the design and implementation of high communication rate BCI system. Meanwhile, existing problems and possible development trends in the future were discussed to promote the development of multi-modal BCI combined with SSVEP.
[1] Wolpaw JR, Birbaumer N, McFarland DJ, et al. Brain–computer interfaces for communication and control [J]. Clinical Neurophysiology, 2002, 113(6): 767-791. [2] Hong KS, Khan MJ. Hybrid brain–computer interface techniques for improved classification accuracy and increased number of commands: a review [J]. Frontiers in Neurorobotics, 2017, 11: 35. [3] 刘洁, 吴慧. 类脑智能研究热点及趋势 [J]. 中国生物医学工程学报, 2021, 40(1): 91-98. [4] 陈小刚, 王毅军. 基于脑电的无创脑机接口研究进展 [J]. 科技导报, 2018, 36(12): 22-30. [5] Choi I, Rhiu I, Lee Y, et al. A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives [J]. PLoS ONE, 2017, 12(4): e0176674. [6] Pfurtscheller G, Allison BZ, Brunner C, et al. The hybrid BCI [J]. Frontiers in Neuroscience, 2010, 4(30):30-30. [7] 郑玉甫, 许敏鹏, 明东. 脑-机接口操控效果差异及其预测研究综述 [J]. 中国生物医学工程学报, 2018, 37(6): 749-755. [8] Nakanishi M, Wang Yijun, Chen Xiaogang, et al. Enhancing detection of SSVEPs for a high-speed brain speller using task-related component analysis [J]. IEEE Transactions on Biomedical Engineering, 2018, 65(1): 104-112. [9] Chen Xiaogang, Wang Yijun, Nakanishi M, et al. High-speed spelling with a noninvasive brain–computer interface [J]. Proceedings of the National Academy of Sciences, National Academy of Sciences, 2015, 112(44): E6058-E6067. [10] Chen Xiaogang, Wang Yijun, Gao Shangkai, et al. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface [J]. Journal of Neural Engineering, 2015, 12(4): 046008. [11] Xu Minpeng, Han Jin, Wang Yijun, et al. Implementing over 100 command codes for a high-speed hybrid brain-computer interface using concurrent P300 and SSVEP features [J]. IEEE Transactions on Biomedical Engineering, 2020, 67(11): 3073-3082. [12] Chen Yonghao, Yang Chen, Chen Xiaogang, et al. A novel training-free recognition method for SSVEP-based BCIs using dynamic window strategy [J]. Journal of Neural Engineering, 2021, 18(3): 036007. [13] Carmona L, Diez PF, Laciar E, et al. Multisensory stimulation and EEG recording below the hair-line: a new paradigm on brain computer interfaces [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(4): 825-831. [14] Deng Can, Tong Jiasen, Deng Xuan, et al. Emotion recognition positively correlates with steady-state visual evoked potential amplitude and alpha entrainment [J]. Neuroscience, 2020, 434: 191-199. [15] Kalaganis FP, Chatzilari E, Nikolopoulos S, et al. A collaborative representation approach to detecting error-related potentials in SSVEP-BCIs [C] //Proceedings of the on Thematic Workshops of ACM Multimedia 2017. New York: ACM, 2017: 262-270. [16] Pfurtscheller G, Neuper C. Motor imagery and direct brain-computer communication [J]. Proceedings of the IEEE, 2001, 89(7): 1123-1134. [17] Allison BZ, Brunner C, Kaiser V, et al. Toward a hybrid brain-computer interface based on imagined movement and visual attention [J]. Journal of Neural Engineering, 2010, 7(2): 26007. [18] Yu Tianyou, Xiao Jun, Wang Fangyi, et al. Enhanced motor imagery training using a hybrid BCI with feedback [J]. IEEE Transactions on Biomedical Engineering, 2015, 62(7): 1706-1717. [19] Brunner C, Allison BZ, Krusienski DJ, et al. Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface [J]. Journal of Neuroscience Methods, 2010, 188(1): 165-173. [20] Ko LW, Komarov O, Lin SC. Enhancing the hybrid BCI performance with the common frequency pattern in dual-channel EEG [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(7): 1360-1369. [21] Cao Lei, Li Jie, Ji Hongfei, et al. A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control [J]. Journal of Neuroscience Methods, 2014, 229: 33-43. [22] Yang Dalin, Nguyen TH, Chung WY. A synchronized hybrid brain-computer interface system for simultaneous detection and classification of fusion EEG signals [J]. Complexity, 2020, 2020: 1-11. [23] Pfurtscheller G, Solis-Escalante T, Ortner R, et al. Self-paced operation of an SSVEP-based orthosis with and without an imagery-based "brain switch": a feasibility study towards a hybrid BCI [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18(4): 409-414. [24] Choi BJ, Jo SH. Hybrid SSVEP/ERD BCI for humanoid navigation [C] //2013 13th International Conference on Control, Automation and Systems (ICCAS 2013). Gwangju: IEEE, 2013: 1641-1645. [25] Savić AM, Malešević NM, Popović MB. Feasibility of a hybrid brain-computer interface for advanced functional electrical therapy [J]. The Scientific World Journal, 2014, 2014: 1-11. [26] Li Wenyu, Duan Feng, Sheng Shili, et al. A human-vehicle collaborative simulated driving system based on hybrid brain–computer interfaces and computer vision [J]. IEEE Transactions on Cognitive and Developmental Systems, 2018, 10(3): 810-822. [27] Yan Nong, Wang Chang, Tao Yi, et al. Quadcopter control system using a hybrid BCI based on offline optimization and enhanced human-machine interaction [J]. IEEE Access, 2020, 8: 1160-1172. [28] Wang Zhihua, Yu Yang, Xu Ming, et al. Towards a hybrid BCI gaming paradigm based on motor imagery and SSVEP [J]. International Journal of Human-Computer Interaction, 2019, 35(3): 197-205. [29] Panicker RC, Puthusserypady S, Sun Ying. An asynchronous P300 BCI with SSVEP-based control state detection [J]. IEEE Transactions on Biomedical Engineering, 2011, 58(6): 1781-1788. [30] Li Yuanqing, Pan Jiahui, Wang Fei, et al. A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control [J]. IEEE Transactions on Biomedical Engineering, 2013, 60(11): 3156-3166. [31] Allison BZ, Jin Jing, Zhang Yu, et al. A four-choice hybrid P300/SSVEP BCI for improved accuracy [J]. Brain-Computer Interfaces, 2014, 1(1): 17-26. [32] Yin Erwei, Zhou Zongtan, Jiang Jun, et al. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm [J]. Journal of Neural Engineering, 2013, 10(2): 026012. [33] Yin Erwei, Zhou Zongtan, Jiang Jun, et al. A speedy hybrid BCI spelling approach combining P300 and SSVEP [J]. IEEE Transactions on Biomedical Engineering, 2014, 61(2): 473-483. [34] Yin Erwei, Zeyl T, Saab R, et al. A hybrid brain–computer interface based on the fusion of P300 and SSVEP scores [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2015, 23(4): 693-701. [35] Wang Minjue, Daly I, Allison BZ, et al. A new hybrid BCI paradigm based on P300 and SSVEP [J]. Journal of Neuroscience Methods, 2015, 244: 16-25. [36] Kapgate D, Kalbande D, Shrawankar U. An optimized facial stimuli paradigm for hybrid SSVEP+P300 brain computer interface [J]. Cognitive Systems Research, 2020, 59: 114-122. [37] Xu Minpeng, Han Jin, Wang Yijun, et al. Optimizing visual comfort and classification accuracy for a hybrid P300-SSVEP brain-computer interface [C] // 2017 8th International IEEE/EMBS Conference on Neural Engineering (NER) . Shanghai: IEEE, 2017: 363-366. [38] Xu Minpeng, Qi Hongzhi, Wan Baikun, et al. A hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature [J]. Journal of Neural Engineering, 2013, 10(2): 026001. [39] Xu Minpeng, Chen Long, Zhang Lixin, et al. A visual parallel-BCI speller based on the time–frequency coding strategy [J]. Journal of Neural Engineering, 2014, 11(2): 026014. [40] Chang MH, Lee JS, Heo J, et al. Eliciting dual-frequency SSVEP using a hybrid SSVEP-P300 BCI [J]. Journal of Neuroscience Methods, 2016, 258: 104-113. [41] Jalilpour S, Sardouie SH, Mijani A. A novel hybrid BCI speller based on RSVP and SSVEP paradigm [J]. Computer Methods and Programs in Biomedicine, 2020, 187: 105326. [42] Lim JH, Lee JH, Hwang HJ, et al. Development of a hybrid mental spelling system combining SSVEP-based brain–computer interface and webcam-based eye tracking [J]. Biomedical Signal Processing and Control, 2015, 21: 99-104. [43] Stawicki P, Gembler F, Rezeika A, et al. A novel hybrid mental spelling application based on eye tracking and SSVEP-based BCI [J]. Brain Sciences, 2017, 7(12): 35. [44] Yao Zhaolin, Ma Xinyao, Wang Yijun, et al. High-speed spelling in virtual reality with sequential hybrid BCIs [J]. IEICE Transactions on Information and Systems, 2018, E101.D(11): 2859-2862. [45] Lin Xinyuan, Chen Zhenyi, Xu Kedi, et al. Development of a high-speed mental spelling system combining eye tracking and SSVEP-based BCI with high scalability [C] // 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) . Berlin: IEEE, 2019: 6318-6322. [46] Mannan MMN, Kamran MA, Kang S, et al. A hybrid speller design using eye tracking and SSVEP brain–computer interface [J]. Sensors, 2020, 20(3): 891. [47] 徐晓英, 随力, 赵彦富, 等. 基于眼电信号的人机交互系统研究进展 [J]. 中国医学物理学杂志, 2020, 37(5): 629-634. [48] Saravanakumar D, Reddy MR. A virtual speller system using SSVEP and electrooculogram [J]. Advanced Engineering Informatics, 2020, 44: 101059. [49] Park S, Cha HS, Im CH. Development of an online home appliance control system using augmented reality and an SSVEP-based brain-computer interface [J]. IEEE Access, 2019, 7: 163604-163614. [50] Zhu Yuanlu, Li Ying, Lu Jinling, et al. A hybrid BCI based on SSVEP and EOG for robotic arm control [J]. Frontiers in Neurorobotics, 2020, 14: 583641. [51] Yang Dalin, Nguyen TH, Chung WY. A bipolar-channel hybrid brain-computer interface system for home automation control utilizing steady-state visually evoked potential and eye-blink signals [J]. Sensors, 2020, 20(19): 5474. [52] Zhou Yajun, He Shenghong, Huang Qiyun, et al. A hybrid asynchronous brain-computer interface combining SSVEP and EOG signals [J]. IEEE Transactions on Biomedical Engineering, 2020, 67(10): 2881-2892. [53] Lin Ke, Chen Xiaogang, Huang Xiaoshan, et al. A hybrid BCI speller based on the combination of EMG envelopes and SSVEP [J]. Applied Informatics, 2015, 2: 1. [54] Lin Ke, Cinetto A, Wang Yijun, et al. An online hybrid BCI system based on SSVEP and EMG [J]. Journal of Neural Engineering, 2016, 13(2): 026020. [55] Rezeika A, Benda M, Stawicki P, et al. 30-targets hybrid BNCI speller based on SSVEP and EMG [C] //2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Miyazaki: IEEE, 2018: 153-158. [56] Chai Xiaoke, Zhang Zhimin, Lu Yangting, et al. A hybrid BCI-based environmental control system using SSVEP and EMG signals [C] // World Congress on Medical Physics and Biomedical Engineering 2018. Singapore: Springer Singapore, 2019, 68/3: 59-63. [57] Chai Xiaoke, Zhang Zhimin, Guan Kai, et al. A hybrid BCI-controlled smart home system combining SSVEP and EMG for individuals with paralysis [J]. Biomedical Signal Processing and Control, 2020, 56: 101687. [58] 周伊婕, 宋西姊, 何峰, 等. 基于脑电的多模态神经功能成像新技术研究进展 [J]. 中国生物医学工程学报, 2020, 39(5): 595-602. [59] Du Pengcheng, Li Jianjun, Yang Sijia, et al. Observing the steady-state visual evoked potentials with a compact quad-channel spin exchange relaxation-free magnetometer [J]. Chinese Physics B, 2019, 28(4): 040702. [60] Ji Hong, Chen Badong, Petro NM, et al. Functional source separation for EEG-fMRI fusion: Application to steady-state visual evoked potentials [J]. Frontiers in Neurorobotics, 2019, 13: 24. [61] Lee HJ, Huang SY, Kuo WJ, et al. Concurrent electrophysiological and hemodynamic measurements of evoked neural oscillations in human visual cortex using sparsely interleaved fast fMRI and EEG [J]. NeuroImage, 2020, 217: 116910. [62] Tomita Y, Vialatte FB, Dreyfus G, et al. Bimodal BCI using simultaneously NIRS and EEG [J]. IEEE transactions on Biomedical Engineering, 2014, 61(4): 1274-1284. [63] 贺庆,郝思聪,司娟宁, 等. 面向脑机接口的脑电采集设备硬件系统综述 [J]. 中国生物医学工程学报, 2020, 39(6): 747-758. [64] 雍颖琼, 张宏江, 程奇峰, 等. 混合脑机接口及其研究进展 [J]. 计算机测量与控制, 2020, 28(9): 9-13,28. [65] 刘政, 何峰, 汤佳贝, 等. 基于深度学习的头皮脑电信息解码研究进展 [J]. 中国生物医学工程学报, 2020, 39(2): 215-228. [66] Won DO, Hwang HJ, Dähne S, et al. Effect of higher frequency on the classification of steady-state visual evoked potentials [J]. Journal of Neural Engineering, 2015, 13(1): 016014. [67] Grichnik R, Benda M, Volosyak I. A VR-based hybrid BCI using SSVEP and gesture input [C] // International Work-Conference on Artificial Neural Networks. Cham: Springer-Verlag, 2019: 418-429. [68] Choi J, Jo S. Application of hybrid brain-computer interface with augmented reality on quadcopter control [C] // 2020 8th International Winter Conference on Brain-Computer Interface (BCI). Gangwon: IEEE, 2020: 1-5. [69] Chen Xiaogang, Hu Nan, Wang Yijun, et al. Validation of a brain-computer interface version of the digit symbol substitution test in healthy subjects [J]. Computers in Biology and Medicine, 2020, 120: 103729.
