Image captioning has acquired substantial attention due to its potential applications in computer vision, natural language processing, and multimedia retrieval. However, it is a challenging task that involves generating descriptive and meaningful captions for images using complex algorithms. While most existing research on image captioning has focused on English or other widely spoken languages, there is a growing need for image captioning in under-resourced languages such as Hindi. Thus, this paper introduces two novel encoder-decoder approaches for performing image captioning in Hindi using approaches involving deep learning. The first approach uses simple convolution-based VGG-16 for image feature extraction, and the sentence embeddings from IndicBERT are utilized as the starting point for the language modeling using the Long-Short Term Memory (LSTM) layer. In the second approach, Swin Transformer does the Image feature map extraction, and the benchmark GPT-2 model performs the language modeling. We evaluate our approach on the Flickr8k Hindi caption dataset, containing over 8000 images and captions, and compare the performance of our two proposed models with the state-of-the-art methods in the literature. Experimental results demonstrate that our proposed approach achieves competitive performance with a BLEU score of 0.72 and a GLEU score of 0.65 on the test data. To further assess and analyze the quality of the generated captions, we conducted manual evaluations using adequacy and fluency as the evaluation criteria to get commendable scores of 3.4 and 3.57, respectively, on a scale of 4. Overall, this research study contributes to the development of natural language processing techniques for Indian languages and has the potential to impact a wide range of applications, such as image retrieval and assistive technologies for visually impaired individuals.