Handwritten Text image Recognition (HTR) in medical profession has become the major challenge due to the complexity of doctors’ handwriting styles and the critical need for precise and efficient text recognition. This study proposes an innovative approach that combines the power of Bidirectional Long Short-Term Memory networks (BLSTM) and Convolutional Neural Networks (CNN) to address these challenges effectively. The proposed model harnesses the bidirectional capabilities of BLSTMs to capture contextual dependencies within doctors’ handwritten notes, enabling it to understand and interpret handwriting more accurately. Additionally, CNNs are employed for feature extraction, enabling the model to recognize salient patterns and representations within handwritten text images. This paper presents comprehensive experiments conducted on a diverse dataset of doctors’ handwritten notes, demonstrating the model's superior performance compared to conventional approaches and outperform the existing work in terms of accuracy. Two different crucial metric word error rate and character error rate are accessed through the combination of BLSTM and CNN that yields state-of-the-art accuracy, robustness to variations in handwriting styles, and remarkable adaptability across various medical document in context of Nepal.