Communication is a fundamental human right, yet individuals with hearing or speech impairments face barriers in education, employment, and social services. Over 2.4 million people in Bangladesh rely on Bangla Sign Language (BdSL) for daily communication, but existing technological solutions remain limited in scope and accessibility. This research proposes a deep learning-based system to translate BdSL into spoken or written Bangla, bridging the communication gap. Unlike previous studies, it successfully detects all Bangla alphabets and digits, generating tokens for future sentence construction using NLP. The system leverages the YOLOv10 model, achieving a mean Average Precision (mAP-50) of 0.99241, demonstrating high accuracy in real-time BdSL recognition. Additionally, this study addresses key challenges such as class imbalance, dataset diversity, and real-time processing to improve accessibility for the deaf and hard-of-hearing community. By resolving previous misclassifications in underrepresented categories, the model enhances accuracy and reliability, contributing to the development of a practical and scalable solution for real-world deployment.