We present a novel dual-stream deep learning architecture, AcouSem-AFNet, for automated tuberculosis (TB) detection using acoustic analysis of respiratory sounds. The proposed architecture utilizes two complementary pathways to extract distinct semantic and acoustic characteristics essential for identifying TB-related respiratory patterns. Specifically, the semantic stream employs a Whisper encoder to model structured patterns in respiratory events, while the acoustic stream leverages WavLM to capture detailed temporal dynamics characteristic of TB cough sounds. These distinct features are fused through a specialized backbone with squeeze-excitation mechanisms and residual connections, designed explicitly to maintain discriminative capabilities and mitigate overfitting challenges typical of limited medical datasets. Evaluated on the CODA-TB challenge dataset, our approach achieves state-of-the-art performance with an accuracy of 78.10% and an AUC of 0.79, demonstrating improvements of 3% in AUC and 2% in accuracy over leading baseline methods. Our framework enables rapid, non-invasive TB screening, particularly beneficial for resource-limited settings, demonstrating the feasibility of deep learning-based acoustic analysis as a scalable, preliminary diagnostic tool to enhance global TB screening accessibility. The code and models are publicly available at https://github.com/IAB-IITJ/AcouSem-AFNet