In order to complete the task of captioning images, this paper compares various vision encoder architectures. Along with a common decoder architecture, LSTM, five well-known vision encoders–VGG16, MobileNet, DenseNet, Xception, and Vision Transformer (ViT)–are evaluated. The quality of generated captions is measured by the BLEU-1 score metric, which is used to evaluate each model’s performance. According to our findings, Vision Transformer (ViT) performs better than the other models, with a BLEU-1 score of 0.6, which is the highest. We explore the mechanisms behind ViT’s superior performance through a thorough analysis, including its attention mechanism and capacity to detect long-range dependencies within images. Furthermore, directions for future research are suggested to improve image captioning, including ensemble approaches and the incorporation of transformer-based decoders. This work provides important information about how well various vision encoder architectures perform image captioning tasks, which will direct further research in this area.