Bangla Sign Language (BSL) serves as a vital communication tool for the Bangla-speaking community with hearing and speech impairments. However, current recognition systems struggle with challenges arising from the complexity of Bangla script, the diversity of finger-spelling gestures, and limited adaptability to real-world scenarios. This study introduces a hybrid BSL recognition system that integrates handcrafted features with deep learning approaches to overcome these limitations. Key innovations include trigger mechanisms for compound character recognition, a robust character taxonomy, and a hybrid feature fusion strategy to enhance accuracy and robustness. Using the BdSL36 dataset, which contains over four million annotated images with diverse backgrounds, the system achieves a recognition accuracy of 94% and a mean Average Precision (mAP) of 96.4%. Additionally, it offers real-time performance, with an average gesture recognition time of 1.32 s, making it practical for real-world applications. This research marks a significant advancement in reducing communication barriers for the Bangla-speaking hearing-impaired community, with promising implications for assistive technologies and education.