Overlapping Speech Detection (OSD), which involves identifying segments where multiple speakers talk simultaneously, is essential for tasks such as speaker identification, diarization, and automatic speech recognition. However, existing methods often struggle, especially under real-world acoustic conditions that include noise, reverberation, and emotional fluctuations. This paper introduces TOSD-Net, a robust frame-level OSD approach designed to operate effectively in diverse acoustic environments. TOSD-Net incorporates a local feature extraction module based on 1D Convolutional Neural Network (CNNs), which captures short-term temporal patterns across neighboring frames and learns high-level spectral representations by integrating information across frequency bins. These local features are then passed to a transformer encoder, which models global temporal dependencies by enabling each frame to attend to all others via multi-head self-attention, effectively capturing long-range interactions across the entire sequence. A final classification layer predicts whether each frame corresponds to overlapping or single-speaker speech. To evaluate the proposed approach, overlapping speech segments are simulated using the GRID corpus for neutral speech and the RAVDESS corpus for emotional speech. Both datasets are further augmented with ambient noise, reverberation, and their combinations at multiple signal-to-noise ratios, reflecting a wide range of real-world acoustic conditions. Experimental results show that TOSD-Net significantly outperforms state-of-the-art OSD baselines, demonstrating its effectiveness and robustness across diverse acoustic environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TOSD-Net: A CNN-Transformer Architecture for Robust Frame-Level Overlapping Speech Detection in Diverse Acoustic Conditions

  • Yassin Terraf,
  • Youssef Iraqi

摘要

Overlapping Speech Detection (OSD), which involves identifying segments where multiple speakers talk simultaneously, is essential for tasks such as speaker identification, diarization, and automatic speech recognition. However, existing methods often struggle, especially under real-world acoustic conditions that include noise, reverberation, and emotional fluctuations. This paper introduces TOSD-Net, a robust frame-level OSD approach designed to operate effectively in diverse acoustic environments. TOSD-Net incorporates a local feature extraction module based on 1D Convolutional Neural Network (CNNs), which captures short-term temporal patterns across neighboring frames and learns high-level spectral representations by integrating information across frequency bins. These local features are then passed to a transformer encoder, which models global temporal dependencies by enabling each frame to attend to all others via multi-head self-attention, effectively capturing long-range interactions across the entire sequence. A final classification layer predicts whether each frame corresponds to overlapping or single-speaker speech. To evaluate the proposed approach, overlapping speech segments are simulated using the GRID corpus for neutral speech and the RAVDESS corpus for emotional speech. Both datasets are further augmented with ambient noise, reverberation, and their combinations at multiple signal-to-noise ratios, reflecting a wide range of real-world acoustic conditions. Experimental results show that TOSD-Net significantly outperforms state-of-the-art OSD baselines, demonstrating its effectiveness and robustness across diverse acoustic environments.