Transformer-based models for image captioning have successfully generated descriptive text from visual input. They generally do not capture details of fine-grained objects, especially when the scene is complex and cluttered. Therefore, we will propose an improved framework of image captioning based on YOLOv5 feature fusion. Such an object detection model brings out highly precise features at the object level. These are combined with the global visual features obtained through a transformer-based encoder. The fusion improves the saliency of objects within the model and the spatial relations among them, further enhancing its captions. The following experiment was performed on the benchmark dataset: Flickr8k. The following proposed model achieved remarkable BLEU scores: 0.798 for BLEU-1, 0.640 for BLEU-2, 0.525 for BLEU-3, and 0.418 for BLEU-4, and hence was used for the image captioning tasks over the Flickr8k dataset. The approach outperforms the state-of-the-art baseline transformer-based captioning models regarding BLEU 1–4 scores. The proposed enhancement technique of feature fusion based on YOLOv5 has more potential for generating accurate images in captions according to the context.