A Review of Research Progress of Hybrid Brain-Computer Interface
Shi Wenqiang1, Xiao Xiaolin1, Liu Shuang1, Xu Minpeng1,2, He Feng1,2*, Ming Dong1,2#
1(School of Precision Instrument and Opto-electronics Engineering, Tianjin University, Tianjin 300072, China) 2(Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China)
Abstract：Traditional brain-computerinterface (BCI) has many shortcomings in practical applications, such as a small instruction set, a small range of people, hard to achieve multi-dimensional control and asynchronous control. Hybrid Brain-Computer Interface (hBCI) can effectively solve these problems. In this paper, three common types of hBCI were reviewed, including hBCI based on EEG signals, hBCI based on EEG signals and other brain signals, and hBCI based on multiple physiological signals. In addition, this paper focused on the research status of hBCI systems and analyzed the stimulus paradigm, control strategy, classification performance, and practical application. The analysis results showed that compared with the traditional BCI system, the hBCI system has a much larger instruction set and higher accuracy. Moreover, due to the combination of other brain signals or physiological signals, hBCI is easier to realize multi-dimensional control and asynchronous control and has achieved rapid development in the utility performance of the system. Finally, this paper summarized different types of hBCI systems and proposed existing problems and future development prospects of hBCI.
施文强, 肖晓琳, 刘爽, 许敏鹏, 何峰, 明东. 混合范式脑-机接口研究进展综述[J]. 中国生物医学工程学报, 2022, 41(1): 73-85.
Shi Wenqiang, Xiao Xiaolin, Liu Shuang, Xu Minpeng, He Feng, Ming Dong. A Review of Research Progress of Hybrid Brain-Computer Interface. Chinese Journal of Biomedical Engineering, 2022, 41(1): 73-85.
[1] Wolpaw JR, Millan JDR, Ramsey NF. Brain-computer interfaces: definitions and principles [J]. Handb Clin Neurol, 2020, 168: 15-23. [2] Ravindran PV, Vinod AP. Name Familiarity Detection using EEG-based Brain Computer Interface [C] // TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON). Kochi: IEEE, 2019: 2305-2309. [3] Lee J, Mukae N, Arata J, et al. Comparison of feature vector compositions to enhance the performance of NIRS-BCI-triggered robotic hand orthosis for post-stroke motor recovery [J]. Applied Sciences-Basel, 2019, 9(18): 1-14. [4] Sorger B; Goebel R. Real-time fMRI for brain-computer interfacing [J]. Handbook of Clinical Neurology, 2020 168: 289-302. [5] Roy S, Rathee D, McCreadie K, et al. Channel selection improves MEG-based brain-computer interface [C] // The 9th IEEE/EMBS International Conference on Neural Engineering (NER). San Francisco: IEEE, 2019: 295-298. [6] 陈小刚, 王毅军. 基于脑电的无创脑机接口研究进展 [J]. 科技导报, 2018, 36(12):22-30. [7] Tang JB, Xu MP, Han J, et al. Optimizing SSVEP-based BCI system towards practical high-speed spelling [J]. Sensors, 2020, 20(15): 1-17. [8] Eidel M, Kubler A. Wheelchair control in a virtual environment by healthy participants using a P300-BCI based on tactile stimulation: training effects and usability [J]. Frontiers in Human Neuroscience, 2020, 14: 1-11. [9] Wang Hongtao, Li Ting, Bezerianos A, et al. The control of a virtual automatic car based on multiple patterns of motor imagery BCI [J]. Medical & Biological Engineering & Computing, 2019, 57(1): 299-309. [10] 李鹏海, 许敏鹏, 万柏坤, 等. 视觉诱发电位脑-机接口实验范式研究进展 [J]. 仪器仪表学报, 2016, 37(10):2340-2351. [11] Ramadan RA, Vasilakos AV. Brain computer interface: control signals review [J]. Neurocomputing, 2017, 223: 26-44. [12] Gao Qiang, Dou Lixiang, Belkacem AN, et al. Noninvasive electroencephalogram based control of a robotic arm for writing task using hybrid BCI system [J]. Biomed Res Int, 2017, 2017:1-8. [13] 钱久超, 夏斌, 杨文璐. 基于P300诱发电位的脑机接口技术研究综述[C]// 全国第21届计算机技术与应用(CACIS)学术会议. 上海: CACIS, 2010: 80-87. [14] 赵欣, 陈志堂, 王坤, 等. 运动想象脑-机接口新进展与发展趋势 [J]. 中国生物医学工程学报, 2019, 38(1):84-93. [15] Katyal A, Singla R. SSVEP-P300 hybrid paradigm optimization for enhanced information transfer rate [J]. Biomedical Engineering Applications Basis and Communications, 2020, 32(1): 1-12. [16] Li Jie, Ji Hongfei, Cao Lei, et al. Evaluation and application of a hybrid brain computer interface for real wheelchair parallel control with multi-degree of freedom [J]. Int J Neural Syst, 2014, 24(4): 1-15. [17] McGeady C, Vukovic A, Puthusserypady S, et al. A hybrid MI-SSVEP based brain computer interface for potential upper limb neurorehabilitation: a pilot study [C] // The 7th International Winter Conference on Brain-Computer Interface (BCI). Gangwon: IEEE, 2019: 106-111. [18] Yu Yang, Zhou Zongtan, Liu Yadong, et al. Self-paced operation of a wheelchair based on a hybrid brain-computer interface combining motor imagery and P300 potential [J]. IEEE Trans Neural Syst Rehabil Eng, 2017, 25(12): 2516-2526. [19] Brumberg JS, Pitt KM, Burnison JD. A noninvasive brain-computer interface for real-time speech synthesis: the importance of multimodal feedback [J]. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(4): 874-881. [20] Zhang Xiangzi, Guo Yaqiu, Gao Boyu, et al. Alpha frequency intervention by electrical stimulation to improve performance in Mu-based BCI [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(6):1262-1270. [21] Choi I, Kwon G H, Lee S, et al. Functional electrical stimulation controlled by motor imagery brain-computer interface for rehabilitation [J]. Brain Sci, 2020, 10(8):512. [22] Zhou YJ, He SH, Huang QY, et al. A hybrid asynchronous brain-computer interface combining SSVEP and EOG signals [J]. IEEE Transactions on Biomedical Engineering, 2020, 67(10): 2881-2892. [23] Leeb R, Sagha H, Chavarriaga R, et al. A hybrid brain-computer interface based on the fusion of electroencephalographic and electromyographic activities [J]. J Neural Eng, 2011, 8(2): 1-6. [24] Hong KS, Khan MJ. Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: a review [J]. Front Neurorobot, 2017, 11(35): 1-27. [25] Allison BZ, Brunner C, Kaiser V, et al. Toward a hybrid brain-computer interface based on imagined movement and visual attention [J]. J Neural Eng, 2010, 7(2): 1-9. [26] Pfurtscheller G, Solis ET, Ortner R, et al. Self-paced operation of an SSVEP-Based orthosis with and without an imagery-based "brain switch": a feasibility study towards a hybrid BCI [J]. IEEE Trans Neural Syst Rehabil Eng, 2010, 18(4): 409-414. [27] Horki P, Solis ET, Neuper C, et al. Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb [J]. Med Biol Eng Comput, 2011, 49(5): 567-577. [28] Cao Lei, Li Jie, Ji Hongfei, et al. A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control [J]. Journal of Neuroscience Methods, 2014, 229: 33-43. [29] Ianosi ADA, Mandru DS, Abrudean A. A hybrid brain-computer interface implementation for the control of an assistive device [C]//2017 E-Health and Bioengineering Conference (EHB). Sinaia: IEEE, 2017: 543-546. [30] Duan Feng, Lin Dongxue, Li Wenyu, et al. Design of a multimodal EEG-based Hybrid BCI system with visual servo module [J]. IEEE Transactions on Autonomous Mental Development, 2015, 7(4): 332-341. [31] Wang Zihua, Yu Yang, Xu Ming, et al. Towards a hybrid BCI gaming paradigm based on motor imagery and SSVEP [J]. International Journal of Human–Computer Interaction, 2019, 35(3): 197-205. [32] Yang Dalin, Nguyen TH, Chung Wan-Young. A synchronized hybrid brain-computer interface system forsimultaneous detection and classification of fusion EEG signals [J]. Hindawi Complexity, 2020, 2020: 1-11. [33] Panicker RC, Puthusserypady S, Sun Ying. An asynchronous P300 BCI with SSVEP-based control state detection [J]. IEEE Trans Biomed Eng, 2011, 58(6): 1781-1788. [34] Yin Erwei, Zhou Zongtan, Jiang Jun, et al. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm [J]. J Neural Eng, 2013, 10(2): 026012. [35] Xu Minpeng, Qi Hongzhi, Wan Baikun, et al. A hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature [J]. J Neural Eng, 2013, 10(2): 026001. [36] Bi Luzheng, Lian Jinling, Jie Ke, et al. A speed and direction-based cursor control system with P300 and SSVEP [J]. Biomedical Signal Processing and Control, 2014, 14: 126-133. [37] Xu Minpeng, Chen Long, Zhang Lixin, et al. A visual parallel-BCI speller based on the time-frequency coding strategy [J]. J Neural Eng, 2014, 11(2): 026014. [38] Fan Xinan, Bi Luzheng, Teng Teng, et al. A brain-computer interface-based vehicle destination selection system using P300 and SSVEP signals [J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(1): 274-283. [39] Xu Minpeng, Han Jin, Wang Yijun, et al. Implementing over 100 command codes for a high-speed hybrid brain-computer interface using concurrent P300 and SSVEP features [J]. Transactions on Biomedical Engineering, 2020, 67(11): 3073-3082. [40] Long Jinyi, Li Yuanqing, Yu Tianyou, et al. Target selection with hybrid feature for BCI-based 2-D cursor control [J]. IEEE Trans Biomed Eng, 2012, 59(1): 132-140. [41] Bhattacharyya S, Konar A,Tibarewala DN. Motor imagery, P300 and error-related EEG-based robot arm movement control for rehabilitation purpose [J]. Med Biol Eng Comput, 2014, 52(12): 1007-1017. [42] Bai Lijuan, Yu Tianyou, Li Yuanqing. A brain computer interface-based explorer [J]. J Neurosci Methods, 2015, 244: 2-7. [43] Zuo Cili, Jin Jing, Yin Erwei, et al. Novel hybrid brain-computer interface system based on motor imagery and P300 [J]. Cogn Neurodyn, 2020, 14(2): 253-265. [44] 周鹏, 曹红宝, 熊屹, 等. 基于脑机接口的智能康复系统的设计 [J]. 计算机工程与应用, 2007, 26: 1-4. [45] Ahn S, Ahn M, Cho H, et al. Achieving a hybrid brain-computer interface with tactile selective attention and motor imagery [J]. J Neural Eng, 2014, 11(6): 066004. [46] Cui Gaochao, Zhao Qibin, Cao Jianting, et al. Hybrid-BCI: classification of auditory and visual related potentials [C] // International Symposium on Soft Computing & Intelligent Systems. Kitakyushu: IEEE, 2015: 297-300. [47] Barbosa S, Pires G, Nunes U. Toward a reliable gaze-independent hybrid BCI combining visual and natural auditory stimuli [J]. J Neurosci Methods, 2016, 261: 47-61. [48] Bhattacharyya S, Clerc M, Hayashibe M. Augmenting motor imagery learning for brain–computer interfacing using electrical stimulation as feedback [J]. IEEE Transactions on Medical Robotics and Bionics, 2019, 1(4): 247-255. [49] Yi Weibo, Qiu Shuang, Wang Kun, et al. Enhancing performance of a motor imagery based brain-computer interface by incorporating electrical stimulation-induced SSSEP [J]. J Neural Eng, 2017, 14(2): 026002. [50] Corsi MC, Chavez M, Schwartz D, et al. Integrating EEG and MEG signals to improve motor imagery classification in brain-computer interface [J]. International Journal of Neural Systems, 2019, 29(1): 1850014. [51] Pfurtscheller G, Allison BZ, Brunner C, et al. The hybrid BCI [J]. Front Neurosci, 2010, 4(30): 1-11. [52] Siamac F, Jan M, Jens S, et al. Enhanced performance by a hybrid NIRS-EEG brain computer interface [J]. NeuroImage, 2012, 59(1): 519-529. [53] Lee MH, Fazli S, Mehnert J, et al. Hybrid brain-computer interface based on EEG and NIRS modalities [C] // International Winter Workshop on Brain-computer Interface. Jeongsun-kun: IEEE, 2014: 1-2. [54] Jawad KM, Jiyoun HM, Keum SH. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface [J]. Frontiers in Human Neuroence, 2014, 8(1): 1-10. [55] Almajidy RK, Boudria Y, Hofmann UG, et al. Multimodal 2D brain computer interface [C] // The 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Milan: IEEE, 2015: 1067-1070. [56] Yin Xuxian, Xu Baolei, Jiang Changhao, et al. A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching [J]. Journal of Neural Engineering, 2015, 12(3): 036004. [57] Shin J, Kim DW, Muller KR, et al. Improvement of information transfer rates using a hybrid eeg-nirs brain-computer interface with a short trial length: offline and pseudo-online analyses [J]. Sensors (Basel), 2018, 18(6):1-16. [58] Kwon J, Shin J, Im CH. Toward a compact hybrid brain-computer interface (BCI): performance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels [J]. PLoS ONE, 2020, 15(3): e0230491. [59] Carlson T, Tonin L, Perdikis S, et al. A hybrid BCI for enhanced control of a telepresence robot [C] // Engineering in Medicine & Biology Society. Osaka: IEEE, 2013: 3097-3100. [60] 谢平, 陈迎亚, 郝艳彪, 等. 基于脑肌电融合的混合脑机接口研究 [J]. 中国生物医学工程学报, 2016, 35(1): 20-30. [61] Ke, Lin, Andrea C, Wang Yijun, et al. An online hybrid BCI system based on SSVEP and EMG [J]. Journal of Neural Engineering, 2016, 13(2): 1-16. [62] Jie Hong. Multimodal brain-computer interface combining synchronously electroencephalography and electromyography [J]. Journal of Intelligent & Fuzzy Systems, 2017, 33(6): 3355-3362. [63] Chai Xiaoke, Zhang Zhimin, Guan Kai, et al. A hybrid BCI-controlled smart home system combining SSVEP and EMG for individuals with paralysis [J]. Biomedical Signal Processing and Control, 2020, 56:101687. [64] Usakli AB, Gurkan S, Aloise F, et al. A hybrid platform based on EOG and EEG signals to restore communication for patients afflicted with progressive motor neuron diseases [C] // 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Minneapolis: IEEE, 2009: 543-546. [65] Jiang J, Zhou Z, Yin E, et al. Hybrid brain-computer interface (BCI) based on the EEG and EOG signals [J]. Biomedical Materials and Engineering, 2014, 24(6): 2919-2925. [66] Soekadar SR, Witkowski M, Gómez C, et al. Hybrid EEG/EOG-based brain/neural hand exoskeleton restores fully independent daily living activities after quadriplegia [J]. Science Robotics, 2016, 1(1): 1-8. [67] Piotr S, Felix G, Aya R, et al. A novel hybrid mental spelling application based on eye tracking and SSVEP-based BCI [J]. Brain Sciences, 2017, 7(4): 1-19. [68] Lee MH, Williamson J, Won DO, et al. A high performance spelling system based on EEG-EOG signals with visual feedback [J]. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(7): 1443-1459. [69] Huang Qiyun, Zhang Zhijun, Yu Tianyou, et al. An EEG-/EOG-based hybrid brain-computer interface: application on controlling an integrated wheelchair robotic arm system [J]. Front Neurosci, 2019, 13:1243. [70] Chen Chao, Zhou Peng, Belkacem AN, et al. Quadcopter robot control based on hybrid brain-computer interface system [J]. Sensors and Materials, 2020, 32(3):991-1004. [71] He Shenghong, Zhou Yajun, Yu Tianyou, et al. EEG-and EOG-based asynchronous hybrid BCI: a system integrating a speller, a web browser, an e-mail client, and a file explorer [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(2): 519-530. [72] Allison BZ, Leeb R, Brunner C, et al. Toward smarter BCIs: extending BCIs through hybridization and intelligent control [J]. Journal of Neural Engineering, 2012, 9(1): 013001. [73] Muller GR, Breitwieser C, Cincotti F, et al. Tools for brain-computer interaction: a general concept for a hybrid BCI [J]. Frontiers in Neuroinformatics, 2011, 5(4): 30. [74] 胡良梅, 高隽. 信息融合技术 [C] // 全国计算机科学及其在仪器仪表中的应用学术交流会. 深圳: 中国仪器仪表学会, 2001: 6-9. [75] Sharma R, Pavlovic VI, Huang TS. Toward multimodal human-computer interface [J]. Proceedings of the IEEE, 1998, 86(5): 853-869. [76] 匡光涛, 王索刚. 一种基于SSVEP和P300的联合脑-机接口范式[J]. 中国科技论文在线, 2014, 2014:1-8. [77] Chen Xiaogang, Chen Zhikai, Gao Shangkai, et al. A high-ITR SSVEP-based BCI speller [J]. Brain-Computer Interfaces, 2014, 1(3-4):181-191. [78] 蒲江波, 安兴伟, 李杰威, 等. 基于体感电刺激的脑－机接口实验范式初探 [J]. 纳米技术与精密工程, 2015, 13(5): 376-382. [79] 明东. 用于脑机接口的感觉刺激事件相关电位研究进展 [J]. 电子测量与仪器学报, 2009, 23(6): 1-6. [80] Stefanics G, Stavrinou M, Sestieri C, et al. Cross-modal visual-auditory-somato sensory integration in a multimodal object recognition task in humans [J]. International Congress Series, 2005,1278(10) :163-166. [81] An Xingwei, Hohne J, Ming Dong, et al. Exploring combinations of auditory and visual stimuli for gaze-independent brain-computer interfaces [J]. PLoS ONE, 2014, 9(10): e111070. [82] 韩向可, 郭士杰. 基于SSVEP_SSA融合的混合脑机接口研究 [J]. 仪器仪表学报, 2019, 40(5): 213-220. [83] Li JN, Pu JB, Cui HY, et al. An Online P300 Brain-Computer Interface Based on Tactile Selective Attention of Somatosensory Electrical Stimulation [J]. Journal of Medical and Biological Engineering, 2019, 32(5): 732-738. [84] Chowdhury A, Raza H, Meena YK, et al. An EEG-EMG correlation-based brain-computer interface for hand orthosis supported neuro-rehabilitation [J]. Journal of Neuroscience Methods, 2019, 312: 1-11. [85] Shi TW, Wang H, Cui WH, et al. Indoor space target searching based on EEG and EOG for UAV [J]. Soft Computing, 2019, 23(21): 11199-11215. [86] Rosca SD, Leba M. Design of a brain-controlled video game based on a BCI system [C] // The 9th International Conference on Manufacturing Science and Education (MSE 2019): Trends in New Industrial Revolution. Sibiu: MSE,2019: 1-9. [87] Ng DWK, Goh SY. Indirect control of an autonomous wheelchair using SSVEP BCI [J]. Journal of Robotics and Mechatronics, 2020, 32(4): 761-767. [88] Zhu Yulian, Liang Sijie. Review of braincomputer interface technology in the treatment of motor dysfunction after stroke [J]. Rehabilitation Medicine, 2020, 30(2): 162-166. [89] Nagel, S, Spuler M. Asynchronous non-invasive high-speed BCI speller with robust non-control state detection [J]. Scientific Reports, 2019, 2: 1-4. [90] Manolova A, Tsenov G, Lazarova V, et al. Combined EEG and EMG fatigue measurement framework with application to hybrid brain-computer interface [C] // 2016 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom). Varna: IEEE, 2016: 1-5. [91] Yan Mingmin, Go Sou, Tamura H, et al. Communication system using EOG for persons with disabilities and its judgment by EEG [J]. Artificial Life and Robotics, 2013, 19(1): 89-94. [92] Zhang T, Chen C, Nakamura M. Reliable EOG signal-based control approach with EEG signal judgment [J]. Artificial Life and Robotics, 2009, 14(2): 195-198. [93] Eduardo I, Andrés Ù, José MA. Multimodal human-machine interface based on a brain-computer interface and an electrooculography interface [C] // 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Boston: IEEE, 2011:4572-4575. [94] Li Yuanqing, Pan Jiahui, Long Jinyi, et al. Multimodal BCIs: target detection, multidimensional control, and awareness evaluation in patients with disorder of consciousness [J]. Proceedings of the IEEE, 2016, 104(2): 332-352. [95] Singla R, Jatana S. The types of hybrid modalities in brain-computer interface systems: a review [J]. International Journal of Biomedical Engineering and Technology, 2017, 23(1): 48-58. [96] Brunner C, Allison BZ, Altstatter C, et al. A comparison of three brain-computer interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals [J]. Journal of Neural Engineering, 2011, 8(2): 1-7. [97] Yu Tianyou, Li Yuanqing, Long Jinyi, et al. A hybrid brain-computer interface-based mail client [J]. Computational and Mathematical Methods in Medicine, 2013, 2013: 750934. [98] Yu Tianyou, Li Yuanqing, Long Jinyi, et al. Surfing the internet with a BCI mouse [J]. Journal of Neural Engineering, 2012, 9(3): 036012.
