Image captioning is a challenging problem that requires producing coherent textual descriptions for images based on both visual and linguistic information. While English language has seen great progress in image captioning, Hindi caption generation is less explored because of the lack of annotated datasets and complexities of the language. To address this challenge, we have utilized the flickr8K dataset in Hindi which was created via the conversion of the English captions to Hindi using the Google Cloud Translator API, similar to the process followed by Ankit Rathi [5]. The proposed image captioning model follows an encoder-decoder architecture. InceptionV3 and YOLOv8 are employed as feature extractors in the encoder to capture high-level visual representations from images. The features that were extracted then entered the multilayer LSTM decoder with Bahdanau attention to guide the model into focusing on relevant areas of the original image during caption generation. The model’s performance was evaluated on the Hindi version of the flickr8k dataset using the BLEU metric. The evaluated BLEU-1, BLEU-2, BLEU-3, BLEU-4 scores were 0.671, 0.501, 0.342, and 0.232 respectively.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hybrid Hindi Image Captioning Using YOLO with Attention-Based LSTM

  • Shweta Meena,
  • Gurtej Singh,
  • Dilwinder Singh

摘要

Image captioning is a challenging problem that requires producing coherent textual descriptions for images based on both visual and linguistic information. While English language has seen great progress in image captioning, Hindi caption generation is less explored because of the lack of annotated datasets and complexities of the language. To address this challenge, we have utilized the flickr8K dataset in Hindi which was created via the conversion of the English captions to Hindi using the Google Cloud Translator API, similar to the process followed by Ankit Rathi [5]. The proposed image captioning model follows an encoder-decoder architecture. InceptionV3 and YOLOv8 are employed as feature extractors in the encoder to capture high-level visual representations from images. The features that were extracted then entered the multilayer LSTM decoder with Bahdanau attention to guide the model into focusing on relevant areas of the original image during caption generation. The model’s performance was evaluated on the Hindi version of the flickr8k dataset using the BLEU metric. The evaluated BLEU-1, BLEU-2, BLEU-3, BLEU-4 scores were 0.671, 0.501, 0.342, and 0.232 respectively.