From Slices to Volumes: Multi-scale Fusion of 2D and 3D Features for CT Scan Report Generation
摘要
The increasing complexity of medical imaging data underscores the necessity for multimodal intelligent systems capable of integrating diverse data representations for comprehensive and precise analysis. In the domain of 3D CT scans, the generation of accurate and clinically meaningful medical reports requires both volumetric contextual information and the fine-grained spatial details inherent in 2D slices. To address this challenge, we propose a framework that employs a pretrained 2D self-supervised learning encoder, initially trained on CT scan slices integrated with a 3D aggregator. By combining the rich, high-resolution information from 2D slices with the spatial coherence of 3D volumetric data, our approach maximizes the complementary strengths of both representations. Experimental results demonstrate that our method outperforms existing baseline approaches in both report generation and multiple-choice question answering, highlighting the critical role of multidimensional feature integration. This work underscores the transformative potential of multimodal intelligent systems in bridging complex imaging data with practical clinical insights, ultimately improving radiological diagnostics and patient care (Our code is now available at github.com/serag-ai/SAMF ).