CausalMPT: Causal Multimodal Prompt Tuning for Healthcare MIE
摘要
Healthcare and elderly-care domains increasingly demand the integration of heterogeneous multimodal information–ranging from medical images and patient records to health-related social data–to enable accurate and timely decision-making. This necessitates advanced multimodal information extraction (MIE) techniques, yet current multimodal large language models (MLLMs) still face challenges of modal imbalance, spurious correlations, and limited robustness due to their reliance on associative rather than causal learning. To address these challenges, we propose causal multimodal prompt tuning (CausalMPT), explicitly introducing causal reasoning into MLLMS to enhance multimodal understanding. CausalMPT adopts a modular design and features three key innovations: a counterfactual intervention module for eliminating false correlations and revealing causal relationships between modalities; The parameter efficient tuning module based on lora can effectively adapt to large-scale models with limited resources. And a prompt answer module, which uses prompts for specific tasks to guide MIE’s fine-grained causal reasoning. A large number of experiments conducted on four benchmark datasets have shown that CausalMPT has achieved consistent improvements. Compared with the basic model, F1 has increased by \(1.47\%\) , \(0.11\%\) , \(1.20\%\) , and \(5.02\%\) respectively.