HYBRID TRANSFORMER–CNN FRAMEWORK FOR EXPLAINABLE MULTIMODAL BRAIN TUMOR CLASSIFICATION USING MULTI-SEQUENCE MRI
Article Sidebar
Main Article Content
Abstract
There has been significant progress in the field of medical image analysis due to advancements in artificial intelligence (AI) and deep learning technologies, enabling automated, more accurate, and more efficient disease diagnosis. An AI-based model, Hybrid CNN-Transformer, is introduced for explainable brain tumor classification using MRI brain images. The proposed framework leverages the feature-extraction capabilities of CNNs and the context-learning capabilities of Transformers to achieve better diagnostic outcomes and increased interpretability. For this purpose, a freely available MRI brain tumor dataset, comprising glioma, meningioma, pituitary, and no-tumor categories, was used to develop and validate the proposed framework. Prior to developing the model, preprocessing steps, such as resizing, normalization, and data augmentation, were applied to MRI images to enhance image quality and improve the model's generalization capacity. The DenseNet121 CNN was used to extract features in the spatial domain, whereas the Transformer encoder was used to capture contextual relationships via self-attention mechanisms. Experimental evaluation has been conducted using accuracy, precision, recall, F1 score, specificity, the confusion matrix, and ROC-AUC. Comparison with CNN and Transformer models shows that the proposed approach achieves competitive diagnostic accuracy while offering better explainability. The results clearly indicate considerable promise in using hybrid architectures for AI-based radiological diagnostics
Downloads
Article Details
Issue
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How to Cite
References
1. Mohamed, E., Gaber, T., Karam, O., & Rashed, E. A. (2022). A Novel CNN pooling layer for breast cancer segmentation and classification from thermograms. Plos one, 17(10), e0276523. https://doi.org/10.1371/journal.pone.0276523 DOI: https://doi.org/10.1371/journal.pone.0276523
2. Alruwaili, A. J. A., Alhawiti, S. A. A., Alhawiti, R. A. H., ALshammari, S. S. R., Aldhuwalia, Y. I., ALOTAIBI, H. A. N., & saleh Alzaid, H. (2024). Advances in Medical Imaging for Early Diagnosis and Disease Management. Saudi Journal of Medicine and Public Health, 1(2), 1584-1595. https://doi.org/10.64483/202412445 DOI: https://doi.org/10.64483/202412445
3. Al-Shoukry, S., & Musa, Z. B. (2024). An overview on effectiveness of activation functions in processing medical images. Al-Furat Journal of Innovations in Electronics and Computer Engineering, 3(2), 163-180. https://doi.org/10.46649/fjiece.v3.2.13a.22.5.2024 DOI: https://doi.org/10.46649/fjiece.v3.2.13a.22.5.2024
4. Applications of Transformers in medical image analysis: https://www.researchgate.net/figure/Applications-of-Transformers-in-medical-image-analysis-as-reviewed-in-this-work_fig5_358846126
5. Erben, A., Mayer, R., & Jacobsen, H. A. (2024). How can we train deep learning models across clouds and continents? An experimental study. Proceedings of the VLDB Endowment, 17(6), 1214–1226. DOI: https://doi.org/10.14778/3648160.3648165
6. Fontes, M., De Almeida, J. D. S., & Cunha, A. (2024). Application of example-based explainable artificial intelligence (XAI) for analysis and interpretation of medical imaging: a systematic review. IEEE Access, 12, 26419-26427. Digital Object Identifier 10.1109/ACCESS.2024.3367606 DOI: https://doi.org/10.1109/ACCESS.2024.3367606
7. GeeksforGeeks (2017). Introduction to Convolution Neural Network. [online] GeeksforGeeks. Available at: https://www.geeksforgeeks.org/machine-learning/introduction-convolution-neural-network/.
8. Nazir, M. I., Akter, A., Wadud, M. A. H., & Uddin, M. A. (2024). Utilizing customized CNN for brain tumor prediction with explainable AI. Heliyon, 10(20), e38997. DOI: https://doi.org/10.1016/j.heliyon.2024.e38997
9. Guluwadi, S. (2024). Enhancing brain tumor detection in MRI images through explainable AI using Grad-CAM with Resnet 50. BMC Medical Imaging, 24(1), 1-19. https://doi.org/10.1186/s12880-024-01292-7 DOI: https://doi.org/10.1186/s12880-024-01292-7
10. Guo, S., Wang, L., Chen, Q., Wang, L., Zhang, J., & Zhu, Y. (2022). Multimodal MRI image decision fusion-based network for glioma classification. Frontiers in oncology, 12, 819673. https://doi.org/10.3389/fonc.2022.819673 DOI: https://doi.org/10.3389/fonc.2022.819673
11. Henry, E. U., Emebob, O., & Omonhinmin, C. A. (2022). Vision transformers in medical imaging: A review. arXiv preprint arXiv:2211.10043. https://arxiv.org/pdf/2211.10043
12. Hou, Y., Wu, Z., Cai, X., & Zhu, T. (2024). The application of an improved DenseNet algorithm in accurate image recognition. Scientific Reports, 14(1), 8645. https://doi.org/10.1038/s41598-024-58421-z DOI: https://doi.org/10.1038/s41598-024-58421-z
13. Jiang, X., Hu, Z., Wang, S., & Zhang, Y. (2023). Deep learning for medical image-based cancer diagnosis. Cancers, 15(14), 3608. https://doi.org/10.3390/cancers15143608 DOI: https://doi.org/10.3390/cancers15143608
14. brain-tumor-mri-dataset (2024). Zenodo, Version v2. https://doi.org/10.5281/zenodo.12735702 — this 2024 dataset record explicitly identifies the source as the Brain Tumor MRI Dataset from Kaggle.
15. Khalifa, M., & Albadawy, M. (2024). AI in diagnostic imaging: revolutionising accuracy and efficiency. Computer Methods and programs in biomedicine update, 5, 100146. https://doi.org/10.1016/j.cmpbup.2024.100146 DOI: https://doi.org/10.1016/j.cmpbup.2024.100146
16. Kumar, A. (2022). Deep learning for multi-modal medical imaging fusion: Enhancing diagnostic accuracy in complex disease detection. Int J Eng Technol Res Manag, 6(11), 183. https://www.researchgate.net/profile/Anil-Kumar-541/publication/389988636_DEEP_LEARNING_FOR_MULTI-MODAL_MEDICAL_IMAGING_FUSION_ENHANCING_DIAGNOSTIC_ACCURACY_IN_COMPLEX_DISEASE_DETECTION/links/67db947ee62c604a0df5f6f4/DEEP-LEARNING-FOR-MULTI-MODAL-MEDICAL-IMAGING-FUSION-ENHANCING-DIAGNOSTIC-ACCURACY-IN-COMPLEX-DISEASE-DETECTION.pdf
17. Liao, L., Li, H., Shang, W., & Ma, L. (2022). An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(3), 1-40. https://doi.org/10.1145/3506695 DOI: https://doi.org/10.1145/3506695
18. Montaha, S., Azam, S., Rafid, A. R. H., Hasan, M. Z., Karim, A., Hasib, K. M., ... & Mannan, Z. I. (2022). MNet-10: A robust shallow convolutional neural network model performing an ablation study on medical images to assess the effectiveness of applying an optimal data augmentation technique. Frontiers in Medicine, 9, 924979. https://doi.org/10.3389/fmed.2022.924979 DOI: https://doi.org/10.3389/fmed.2022.924979
19. Niu, J., Liu, S., Li, H., Zhang, T., & Wang, L. (2023). Grasp Detection Combining Self-Attention with CNN in Complex Scenes. Applied Sciences, 13(17), 9655. https://doi.org/10.3390/app13179655 DOI: https://doi.org/10.3390/app13179655
20. Nouis, S. C., Uren, V., & Jariwala, S. (2025). Evaluating accountability, transparency, and bias in AI-assisted healthcare decision-making: a qualitative study of healthcare professionals’ perspectives in the UK. BMC Medical Ethics, 26(1), 89. https://doi.org/10.1186/s12910-025-01243-z DOI: https://doi.org/10.1186/s12910-025-01243-z
21. Obuchowicz, R., Strzelecki, M., & Piórkowski, A. (2024). Clinical applications of artificial intelligence in medical imaging and image processing—A review. Cancers, 16(10), 1870. https://doi.org/10.3390/cancers16101870 DOI: https://doi.org/10.3390/cancers16101870
22. Oladele, O. K. (2024). AI-powered medical imaging: a comprehensive review of applications, benefits, and challenges. Google Scholar. https://www.researchgate.net/profile/Oluwaseyi-Oladele-3/publication/385074391_AI-Powered_Medical_Imaging_A_Comprehensive_Review_of_Applications_Benefits_and_Challenges/links/6713ddd8069cb92a811d0105/AI-Powered-Medical-Imaging-A-Comprehensive-Review-of-Applications-Benefits-and-Challenges.pdf
23. Papa, L., Russo, P., Amerini, I., & Zhou, L. (2024). A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking. IEEE transactions on pattern analysis and machine intelligence, 46(12), 7682-7700. https://ieeexplore.ieee.org/iel7/34/4359286/10508091.pdf DOI: https://doi.org/10.1109/TPAMI.2024.3392941
24. Qian, J., Li, H., Wang, J., & He, L. (2023). Recent advances in explainable artificial intelligence for magnetic resonance imaging. Diagnostics, 13(9), 1571. https://doi.org/10.3390/diagnostics13091571 DOI: https://doi.org/10.3390/diagnostics13091571
25. Poudyal, A., Devkota, N., & Budha, G. (2024). MRI-based brain tumor classification using transfer learning: A comparative analysis. International Journal of Computer Applications, 186(40), 7–11. DOI: https://doi.org/10.5120/ijca2024923988
26. Krishnan, P. T., Krishnadoss, P., Khandelwal, M., Gupta, D., Nihaal, A., & Kumar, T. S. (2024). Enhancing brain tumor detection in MRI with a rotation invariant Vision Transformer. Frontiers in Neuroinformatics, 18, 1414925. DOI: https://doi.org/10.3389/fninf.2024.1414925
27. Shi, D., Liu, R., Tao, L., & Yuan, C. (2022, May). Heuristic dropout: An efficient regularization method for medical image segmentation models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1101-1105). IEEE. DOI: 10.1109/ICASSP43922.2022.9747409 DOI: https://doi.org/10.1109/ICASSP43922.2022.9747409
28. Tjoa, E., & Guan, C. (2020). A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE transactions on neural networks and learning systems, 32(11), 4793-4813. https://ieeexplore.ieee.org/iel7/5962385/9591206/09233366.pdf DOI: https://doi.org/10.1109/TNNLS.2020.3027314
29. Toh, Y. (2024). Ultrasound versus magnetic resonance imaging as first-line imaging strategies for rotator cuff pathologies: a comprehensive analysis of clinical practices, economic efficiency, and future perspectives. Cureus, 16(4). https://doi.org/10.7759/cureus.59231 DOI: https://doi.org/10.7759/cureus.59231
30. Xia, K., & Wang, J. (2023). Recent advances of transformers in medical image analysis: a comprehensive review. MedComm–Future Medicine, 2(1), e38. DOI: 10.1002/mef2.38 DOI: https://doi.org/10.1002/mef2.38
31. Anghel, C., Grasu, M. C., Anghel, D. A., Rusu-Munteanu, G. I., Dumitru, R. L., & Lupescu, I. G. (2024). Pancreatic adenocarcinoma: Imaging modalities and the role of artificial intelligence in analyzing CT and MRI images. Diagnostics, 14(4), 438. https://doi.org/10.3390/diagnostics14040438 DOI: https://doi.org/10.3390/diagnostics14040438
32. Zhang, C., Deng, X., & Ling, S. H. (2024). Next-gen medical imaging: U-Net evolution and the rise of transformers. Sensors, 24(14), 4668. https://doi.org/10.3390/s24144668 DOI: https://doi.org/10.3390/s24144668
33. Zhao, S. S., Feng, X. L., Hu, Y. C., Han, Y., Tian, Q., Sun, Y. Z., ... & Wang, W. (2020). Better efficacy in differentiating WHO grade II from III oligodendrogliomas with machine learning than radiologists’ reading from conventional T1 contrast-enhanced and fluid-attenuated inversion recovery images. BMC neurology, 20(1), 48. https://doi.org/10.1186/s12883-020-1613-y DOI: https://doi.org/10.1186/s12883-020-1613-y
34. Zhou, S. K., Greenspan, H., Davatzikos, C., Duncan, J. S., Van Ginneken, B., Madabhushi, A., ... & Summers, R. M. (2021). A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE, 109(5), 820-838. doi:10.1109/JPROC.2021.3054390 DOI: https://doi.org/10.1109/JPROC.2021.3054390