Foundation Models and their multimodal extensions have transformed AI, enabling flexible reasoning and generation across diverse modalities such as text, vision, audio, and video. However, their black-box nature, large output spaces, and susceptibility to hallucinations raise concerns about reliability, particularly in high-stakes applications. Conformal Prediction offers a principled, distribution-free framework for uncertainty quantification, but its traditional formulations are challenged by the complexity of FM pipelines, multimodal interactions, and streaming or partially supervised scenarios. This chapter provides a detailed survey of recent efforts to adapt CP to deep neural networks and multimodal foundation models. We review innovations spanning generative modeling, predictive pipelines, reasoning and control, and retrieval-augmented generation, highlighting techniques such as sampling-aware calibration, trust-score conditioning and input-adaptive error control. The chapter also synthesizes design principles for robust conformal prediction in multimodal contexts, including per-modality normalization, dependency-aware calibration, and representative evaluation strategies. We conclude by outlining open research directions that can shape reliable and interpretable uncertainty quantification for the next generation of foundation models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Conformal Prediction in the Age of Multimodal Foundation Models: A Survey

  • Christian Stavan,
  • Kunal Tilaganji,
  • Sai Mathura Krishnan,
  • Sai Srinivas Kancheti,
  • Vineeth N. Balasubramanian

摘要

Foundation Models and their multimodal extensions have transformed AI, enabling flexible reasoning and generation across diverse modalities such as text, vision, audio, and video. However, their black-box nature, large output spaces, and susceptibility to hallucinations raise concerns about reliability, particularly in high-stakes applications. Conformal Prediction offers a principled, distribution-free framework for uncertainty quantification, but its traditional formulations are challenged by the complexity of FM pipelines, multimodal interactions, and streaming or partially supervised scenarios. This chapter provides a detailed survey of recent efforts to adapt CP to deep neural networks and multimodal foundation models. We review innovations spanning generative modeling, predictive pipelines, reasoning and control, and retrieval-augmented generation, highlighting techniques such as sampling-aware calibration, trust-score conditioning and input-adaptive error control. The chapter also synthesizes design principles for robust conformal prediction in multimodal contexts, including per-modality normalization, dependency-aware calibration, and representative evaluation strategies. We conclude by outlining open research directions that can shape reliable and interpretable uncertainty quantification for the next generation of foundation models.