<p>Estimating acoustic context parameters is essential for characterizing acoustic environments, thereby enhancing immersive perception in spatial audio creation and improving speech enhancement and dereverberation algorithms. In this paper, we propose a unified deep learning based framework that estimates various acoustic contexts, including frequency-dependent reverberation time (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(T_{30}\)</EquationSource> <EquationSource Format="MATHML"><math> <msub> <mi>T</mi> <mn>30</mn> </msub> </math></EquationSource> </InlineEquation>), direct-to-reverberant ratio, clarity (<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(C_{50}\)</EquationSource> <EquationSource Format="MATHML"><math> <msub> <mi>C</mi> <mn>50</mn> </msub> </math></EquationSource> </InlineEquation>), room geometry, and sound source orientation from first-order Ambisonics (FOA) speech recordings. Our framework employs a novel feature, termed the <i>Spectro-Spatial Covariance Vector</i> (SSCV), which efficiently represents the temporal, spectral, and spatial information of FOA signals. This feature can be effectively utilized by several deep neural networks as back-ends. Experimental results demonstrate that the proposed framework, which incorporates spatial information derived from FOA recordings, significantly outperforms existing methods based solely on spectral information from single-channel audio, achieving more than a 50% reduction in estimation error across all acoustic context estimation tasks. Additionally, we introduce FOA-Conv3D, a novel back-end network that effectively utilizes the SSCV feature through a 3D convolutional encoder. FOA-Conv3D outperforms currently widely applied deep learning frameworks such as convolutional neural network and recurrent convolutional neural network back-end architectures in acoustic parameter and orientation estimation tasks, exhibiting greater robustness under both pink and babble noise conditions. Finally, ablation studies reveal the relative contributions of spectral, interaural level difference, and interaural phase difference cues within the SSCV representation.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A unified deep learning framework for estimating acoustic context parameters from first order ambisonic speech recordings

  • Hanyu Meng,
  • Jeroen Breebaart,
  • Jeremy Stoddard,
  • Vidhyasaharan Sethu,
  • Eliathamby Ambikairajah

摘要

Estimating acoustic context parameters is essential for characterizing acoustic environments, thereby enhancing immersive perception in spatial audio creation and improving speech enhancement and dereverberation algorithms. In this paper, we propose a unified deep learning based framework that estimates various acoustic contexts, including frequency-dependent reverberation time ( \(T_{30}\) T 30 ), direct-to-reverberant ratio, clarity ( \(C_{50}\) C 50 ), room geometry, and sound source orientation from first-order Ambisonics (FOA) speech recordings. Our framework employs a novel feature, termed the Spectro-Spatial Covariance Vector (SSCV), which efficiently represents the temporal, spectral, and spatial information of FOA signals. This feature can be effectively utilized by several deep neural networks as back-ends. Experimental results demonstrate that the proposed framework, which incorporates spatial information derived from FOA recordings, significantly outperforms existing methods based solely on spectral information from single-channel audio, achieving more than a 50% reduction in estimation error across all acoustic context estimation tasks. Additionally, we introduce FOA-Conv3D, a novel back-end network that effectively utilizes the SSCV feature through a 3D convolutional encoder. FOA-Conv3D outperforms currently widely applied deep learning frameworks such as convolutional neural network and recurrent convolutional neural network back-end architectures in acoustic parameter and orientation estimation tasks, exhibiting greater robustness under both pink and babble noise conditions. Finally, ablation studies reveal the relative contributions of spectral, interaural level difference, and interaural phase difference cues within the SSCV representation.