Image captioning, or the generation of descriptive textual representations of visual information, is a significant challenge in natural language processing and computer vision domains. Though significant amelioration has been made for English captioning systems, low-resource languages like Bangla present severe challenges due to limited datasets, linguistic diversity, and the need to bridge visual perception with appropriate lexical expression. This work introduces a novel approach to Bangla image captioning by combining Vision Transformer (ViT) and Bidirectional Gated Recurrent Unit (BiGRU) with the Bahdanau attention mechanism. In contrast to the traditional captioning models that have depended predominantly on CNN-based architectures for visual feature extraction, our model leverages the global attention mechanism of ViT in capturing overall visual context, coupled with a BiGRU decoder that processes linguistic information in both directions. We evaluate our approach on the Ban-Cap dataset, achieving BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores of 0.5621, 0.3473, 0.2476, and 0.1208, respectively. Comparison with CNN-based approaches indicates both the potential as well as the current limitations of transformer-based models for Bangla caption generation.