In this paper, we propose a training-free framework for generating audio descriptions (ADs) by leveraging large pretrained Video-Language Models (VLMs) and Large Language Models (LLMs) without task-specific fine-tuning. Our method enhances video understanding through a semantic-constrained prompting strategy that incorporates temporally coherent context into VLM inputs, while an adaptive character recognition module ensures consistent identity tracking across frames. By explicitly linking visual character observations to narrative elements, the system produces contextually rich and coherent visual descriptions. Finally, the video captions are then refined into a single, concise audio description sentence through a LLM operating exclusively on text inputs, ensuring clarity, brevity, and narrative cohesion. The experimental evaluation performed on the MAD-eval-Named and TV-AD benchmarks, validates the approach achieving CIDEr scores of 23.2 and 23.4, respectively. Compared to state-of-the-art training-free baselines, our framework consistently yields relative improvements ranging from 3.6% to 8% across multiple evaluation metrics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automatic Audio Description: A Training-Free Approach Using Foundation Models

  • Ruxandra Tapu,
  • Bogdan Mocanu

摘要

In this paper, we propose a training-free framework for generating audio descriptions (ADs) by leveraging large pretrained Video-Language Models (VLMs) and Large Language Models (LLMs) without task-specific fine-tuning. Our method enhances video understanding through a semantic-constrained prompting strategy that incorporates temporally coherent context into VLM inputs, while an adaptive character recognition module ensures consistent identity tracking across frames. By explicitly linking visual character observations to narrative elements, the system produces contextually rich and coherent visual descriptions. Finally, the video captions are then refined into a single, concise audio description sentence through a LLM operating exclusively on text inputs, ensuring clarity, brevity, and narrative cohesion. The experimental evaluation performed on the MAD-eval-Named and TV-AD benchmarks, validates the approach achieving CIDEr scores of 23.2 and 23.4, respectively. Compared to state-of-the-art training-free baselines, our framework consistently yields relative improvements ranging from 3.6% to 8% across multiple evaluation metrics.