Domain Code-switched speech recognition presents unique computational challenges in processing multilingual utterances where speakers alternate between languages, requiring simultaneous modeling of multiple phonological systems with distinct temporal characteristics. Problem Contemporary foundation models like Whisper and XLS-R need billions of parameters and thousands of GPU-hours for training. This creates challenges for deployment in resource-constrained environments and adaptation to specialized linguistic domains. Solution This paper presents MARS-ASR (Multi-Resolution Adaptive Recognition System), addressing both data scarcity and architectural efficiency through three synergistic innovations: (1) a scalable reinforcement learning pipeline for high-quality synthetic code-switched data generation, demonstrating 91.3% parallel efficiency at 64 GPUs and producing 300 h of linguistically validated utterances with 4.1/5.0 human quality ratings–this data augmentation constitutes the primary driver of performance improvement, contributing 64.4% of total WER reduction; (2) a parallel triple-branch Conformer encoder processing acoustic signals at 25ms, 50ms, and 100ms temporal resolutions with GPU-optimized execution achieving 94.2% hardware utilization on NVIDIA A100 accelerators, designed to maximally exploit the generated training data through multi-scale acoustic modeling; and (3) a context-adaptive fusion mechanism with learned gating networks that dynamically allocate computational resources based on acoustic complexity, reducing inference cost by 23.4%. Results Experimental evaluation on Hindi–Marathi code-switched speech demonstrates 16.1% Word Error Rate, representing 38.5% relative improvement over single-resolution baselines and 9.6% improvement over XLS-R 2B while requiring 85% fewer parameters. Critically, ablation studies reveal that while RL-generated synthetic data provide the majority of accuracy gains (64.4% contribution), the specialized multi-resolution architecture amplifies this benefit by 62% compared to single-resolution models trained on identical data, demonstrating that targeted data generation and architectural design must be co-optimized. The complete system achieves training efficiency of 320 GPU-hours (2.25<InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\times$</EquationSource> <EquationSource Format="MATHML"><math> <mo>×</mo> </math></EquationSource> </InlineEquation> reduction versus foundation models) and real-time factor 0.23 for inference (4,348 utterances/GPU-hour throughput). Cross-platform benchmarking demonstrates consistent performance scaling across GPU generations (V100, A100, H100). The system achieves 94.3% language identification accuracy and 90.1% switch-point detection F1-score, establishing new benchmarks for computationally efficient multilingual speech processing.

Scalable RL-based data generation and multi-resolution architecture for code-switched speech recognition: a high-performance computing approach

期刊论文
发表日期：

2026年2月8日

查看全文

Hemant Palivela,
Meera Narvekar

摘要

Domain

Code-switched speech recognition presents unique computational challenges in processing multilingual utterances where speakers alternate between languages, requiring simultaneous modeling of multiple phonological systems with distinct temporal characteristics.

Problem

Contemporary foundation models like Whisper and XLS-R need billions of parameters and thousands of GPU-hours for training. This creates challenges for deployment in resource-constrained environments and adaptation to specialized linguistic domains.

Solution

This paper presents MARS-ASR (Multi-Resolution Adaptive Recognition System), addressing both data scarcity and architectural efficiency through three synergistic innovations: (1) a scalable reinforcement learning pipeline for high-quality synthetic code-switched data generation, demonstrating 91.3% parallel efficiency at 64 GPUs and producing 300 h of linguistically validated utterances with 4.1/5.0 human quality ratings–this data augmentation constitutes the primary driver of performance improvement, contributing 64.4% of total WER reduction; (2) a parallel triple-branch Conformer encoder processing acoustic signals at 25ms, 50ms, and 100ms temporal resolutions with GPU-optimized execution achieving 94.2% hardware utilization on NVIDIA A100 accelerators, designed to maximally exploit the generated training data through multi-scale acoustic modeling; and (3) a context-adaptive fusion mechanism with learned gating networks that dynamically allocate computational resources based on acoustic complexity, reducing inference cost by 23.4%.

Results

Experimental evaluation on Hindi–Marathi code-switched speech demonstrates 16.1% Word Error Rate, representing 38.5% relative improvement over single-resolution baselines and 9.6% improvement over XLS-R 2B while requiring 85% fewer parameters. Critically, ablation studies reveal that while RL-generated synthetic data provide the majority of accuracy gains (64.4% contribution), the specialized multi-resolution architecture amplifies this benefit by 62% compared to single-resolution models trained on identical data, demonstrating that targeted data generation and architectural design must be co-optimized. The complete system achieves training efficiency of 320 GPU-hours (2.25 $\times$ $\times$ reduction versus foundation models) and real-time factor 0.23 for inference (4,348 utterances/GPU-hour throughput). Cross-platform benchmarking demonstrates consistent performance scaling across GPU generations (V100, A100, H100). The system achieves 94.3% language identification accuracy and 90.1% switch-point detection F1-score, establishing new benchmarks for computationally efficient multilingual speech processing.