The current movie sound effect dubbing lacks a comprehensive automated solution. We introduce Foley Agent, an automatic movie sound design and mixing agent driven by large language models (LLMs). It processes the movie in three stages: resource collection, sound synchronization, and mixing. First, it comprehends the content of a video clip by querying a Visual LLM and searching for sound effect resources. Subsequently, it adjusts the sound signal to match the motion in the clips. Finally, it mixes the audios based on reasoning. Our approach results in structured and controllable sound with lossless quality, particularly maintaining spatial awareness in multi-channel audio. We provide insights into the capabilities of LLMs for general tasks. To the best of our knowledge, this is the first attempt to use LLMs for film dubbing and mixing.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Foley Agent: Automatic Sound Design and Mixing Agent for Silent Videos Driven by LLMs

  • Kun Lin,
  • Shiguang Liu

摘要

The current movie sound effect dubbing lacks a comprehensive automated solution. We introduce Foley Agent, an automatic movie sound design and mixing agent driven by large language models (LLMs). It processes the movie in three stages: resource collection, sound synchronization, and mixing. First, it comprehends the content of a video clip by querying a Visual LLM and searching for sound effect resources. Subsequently, it adjusts the sound signal to match the motion in the clips. Finally, it mixes the audios based on reasoning. Our approach results in structured and controllable sound with lossless quality, particularly maintaining spatial awareness in multi-channel audio. We provide insights into the capabilities of LLMs for general tasks. To the best of our knowledge, this is the first attempt to use LLMs for film dubbing and mixing.