基于通道注意力机制和多尺度旋转卷积的Mamba双分支医学图像融合模型

doi:10.3969/j.issn.0258-8021.2026.02.002

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (8908 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要针对Transformer计算复杂度过高的问题,提出了一种基于通道注意力机制和多尺度旋转卷积的Mamba双分支融合网络(ReMFM)。首先,利用通道注意力机制捕捉待融合图像通道间的信息关联;然后,设计了旋转卷积模块,在方向与尺度维度上高效提取跨模态局部结构特征;最后,对Mamba进行改进,设计了注意力状态空间模块,引入非因果建模与单次扫描的全局感知能力,在保证表达力的同时显著降低计算复杂度与冗余。本实验使用的数据来自哈佛脑图谱数据库,包含357对MRI和SPECT图像,其中训练集333对,测试集24对。采用重叠剪裁策略进行数据增强,训练数据统一为120像素×120像素。实验结果表明,ReMFM在梯度、图像特征、杨氏度量、视觉信息保真度及互信息指标上分别达到0.743 8、0.745 7、0.976 7、0.688 4和5.022 6,较7种主流Transformer融合模型提升2.52%、15.92%、2.41%、14.64%和14.29%。本研究模型能够有效突出病变区域的结构信息,同时兼顾边缘纹理,可生成高质量融合图像。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	孔韦韦
	韩尹波
	雷阳
	王宇辰
	周皓晨

关键词 ：医学图像融合, 医学图像, 注意力机制, Mamba

Abstract：To address the high computational complexity of Transformers, a Mamba dual-branch fusion network based on channel attention mechanism and multi-scale rotation convolution (rotation-enhanced Mamba fusion model, ReMFM) was proposed in this work. First, the channel attention mechanism was ued to capture the inter-channel information correlation of the images to be fused. Next, a rotation convolution module was designed to efficiently extract cross-modal local structural features in both direction and scale dimensions. Finally, an improvement was made to Mamba by designing an attention state space module, which introduced non-causal modeling and global perception capability with single-scan processing. These actions significantly reduced computational complexity and redundancy while ensuring expressiveness. The data used in this experiment were obtained from the Harvard Brain Atlas Database, comprising 357 paired MRI and SPECT images, with 333 pairs allocated for training and 24 pairs for testing. Data augmentation was performed using an overlapping cropping strategy, and all training images were standardized to a uniform size of 120 × 120 pixels. Experimental results showed that ReMFM achieved 0.7438, 0.7457, 0.9767, 0.6884, and 5.0226 in gradient, image feature, Yann measure, visual information fidelity, and mutual information, respectively, with improvements of 2.52%, 15.92%, 2.41%, 14.64%, and 14.29% over seven mainstream Transformer-based models. The proposed model effectively highlights the structural information of the lesion regions while preserving edge textures, producing high-quality fused images.

Key words： medical image fusion medical images attention mechanisms Mamba

收稿日期: 2025-11-11

PACS:

R318

基金资助:国家自然科学基金(62471389);陕西省科技厅面上项目(2022JM-369)

通讯作者: ^*E-mail:kwwking@163.com

引用本文:

孔韦韦, 韩尹波, 雷阳, 王宇辰, 周皓晨. 基于通道注意力机制和多尺度旋转卷积的Mamba双分支医学图像融合模型[J]. 中国生物医学工程学报, 2026, 45(2): 141-153.
Kong Weiwei, Han Yinbo, Lei Yang, Wang Yuchen, Zhou Haochen. Mamba Dual-Branch Medical Image Fusion Model Based on Channel Attention Mechanism and Multi-Scale Rotational Convolution. Chinese Journal of Biomedical Engineering, 2026, 45(2): 141-153.

链接本文:

http://cjbme.csbme.org/CN/10.3969/j.issn.0258-8021.2026.02.002 或 http://cjbme.csbme.org/CN/Y2026/V45/I2/141

[1] 陈伟, 孙康康, 李奇轩, 等. 多模态医学图像融合方法的研究进展[J]. 中国辐射卫生, 2023, 32(5): 580-585.
[2] 史韶杰, 韩建宁, 李大威, 等. 基于多尺度曝光融合的医学 CT 图像增强方法[J]. 电子测量技术, 2024, 45(6): 106-111.
[3] Ma J, Ma Y, Li C. Infrared and visible image fusion methods and applications: A survey[J]. Information Fusion, 2019, 45(1): 153-178.
[4] James AP, Dasarathy BV. Medical image fusion: A survey of the state of the art[J]. Information Fusion, 2014, 19(9): 4-19.
[5] Li H, Wu XJ. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[6] Xu H, Ma J, Jiang J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
[7] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30.
[8] Fu Y, Xu TY, Wu XJ, et al. PPT fusion: Pyramid patch transformerfor a case study in image fusion[J/OL]. https://arxiv.org/abs/2107.13967,2022-08-03/2025-11-03.
[9] Wang Z, Chen Y, Shao W, et al. SwinFuse: A residual swin transformer fusion network for infrared and visible images[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71(7): 5016412.
[10] Ma J, Tang L, Fan F, et al. SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217.
[11] Qu L, Liu S, Wang M, et al. Transmef: a transformer-based multi-exposure image fusion framework using self-supervised multi-task learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI Press, 2022, 36(2): 2126-2134.
[12] Zhang Y, Liu H, Hu Q. Transfuse: fusing transformers and cnns for medical image segmentation[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference.Strasbourg: Springer International Publishing, 2021: 14-24.
[13] Tang W, He F, Liu Y. YDTR: infrared and visible image fusion via Y-shape dynamic transformer[J]. IEEE Transactions on Multimedia, 2022, 25(7): 5413-5428.
[14] Tang W, He F, Liu Y. ITFuse: an interactive transformer for infrared and visible image fusion[J]. Pattern Recognition, 2024, 156(12): 110822.
[15] Jian L, Xiong S, Yan H, et al. Rethinking cross-attention for infrared and visible image fusion[J/OL]. https://arxiv.org/abs/2401.11675, 2025-06-18, 2025-11-03.
[16] Yang X, Huo H, Li C, et al. Semantic perceptive infrared and visible image fusion transformer[J]. Pattern Recognition, 2024, 149(5): 110223.
[17] Zhao Y, Zheng Q, Zhu P, et al. TUFusion: a transformer-based universal fusion algorithm for multimodal images[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 34(3): 1712-1725.
[18] Fang L, Hou M, Huang B, et al. DCAFusion: a novel general image fusion framework based on reference image reconstruction and dual-cross attention mechanism[J]. Information Sciences, 2025, 698(4): 121772.
[19] Tang W, He F, Liu Y, et al. MATR: multimodal medical image fusion via multiscale adaptive transformer[J]. IEEE Transactions on Image Processing, 2022, 31(7): 5134-5149.
[20] Gu A, Dao T. Mamba: Linear-time sequence modeling with selective state spaces[C]//Proceedings of the First Conference on Language Modeling. Philadelphia: OpenReview.net, 2024: 1-32.
[21] Liu Y, Tian Y, Zhao Y, et al. Vmamba: Visual state space model[J]. Advances in Neural Information Processing Systems, 2024, 37: 103031-103063.
[22] Yin M, Liu X, Liu Y, et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement, 2018, 68(1): 49-64.
[23] Guo H, Guo Y, Zha Y, et al. Mambairv2: Attentive state space restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2025: 28124-28133.
[24] Xu H, Ma J. EMFusion: An unsupervised enhanced medical image fusion network[J]. Information Fusion, 2021, 76(12): 177-186.
[25] Kingma D P. Adam: A method for stochastic optimization[J/OL]. https://arxiv.org/abs/1412.6980,2017-01-30, 2025-11-03.
[26] Tang W, He F. FATFusion: a functional-anatomical transformer for medical image fusion[J]. Information Processing & Management, 2024, 61(4): 103687.
[27] Xie X, Zhang X, Tang X, et al. MACTFusion: lightweight cross transformer for adaptive multimodal medical image fusion[J]. IEEE Journal of Biomedical and Health Informatics, 2025, 29(5): 3317-3328.
[28] Zhao Z, Deng L, Bai H, et al. Image fusion via vision-language model[J/OL]. https://arxiv.org/abs/2402.02235,2024-07-10, 2025-11-03.
[29] Hossny M, Nahavandi S, Creighton D. Comments on ‘Information measure for performance of image fusion’[J]. Electronics Letters, 2008, 44(18): 1066-1067.
[30] Xydeas CS, Petrovic V. Objective image fusion performance measure[J]. Electronics letters, 2000, 36(4): 308-309.
[31] Zhao J, Laganiere R, Liu Z. Performance assessment of combinative pixel-level image fusion based on an absolute feature measurement[J]. Int J Innov Comput Inf Control, 2007, 3(6): 1433-1447.
[32] Yang C, Zhang JQ, Wang XR, et al. A novel similarity based quality metric for image fusion[J]. Information Fusion, 2008, 9(2): 156-160.
[33] Chen Y, Blum RS. A new automated quality assessment algorithm for image fusion[J]. Image and Vision Computing, 2009, 27(10): 1421-1432.
[34] Sheikh HR, Bovik AC. Image information and visual quality[J]. IEEE Transactions on Image Processing, 2006, 15(2): 430-444.
[35] Peng H, Long F, Ding C. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(8): 1226-1238.
[36] Bekkers E J, Lafarge M W, Veta M, et al. Roto-translation covariant convolutional networks for medical image analysis[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada: Springer International Publishing, 2018: 440-448.
[37] Hu Q, Peng Y. Infrared and visible image fusion using a state-space adversarial model with cross-Modal dependency learning[J]. Mathematics, 2025, 13(15): 2333.
[38] Hasan BT, Al-Sabaawi AMA, Al-Araji ZJ. Lightweight deep learning for IoT-based medical diagnosis: a survey[J]. Mesopotamian Journal of Artificial Intelligence in Healthcare, 2025, 2025: 244-259.