Automatic image caption generation using deep learning is rapidly bridging computer vision with natural language processing. Envision a system that might be able to carefully study a picture and then describe it in natural language. This technology has huge potential for totally revolutionizing the way interactions with images are undertaken, greatly improving accessibility for visually challenged users, more accurate image search, and streamlined image organization. It makes use of Vision Transformers, especially the ViT-GPT2 model for the extraction of image features from Flickr8k dataset. Features extracted in this stage are further fed into the Vision Encoder Decoder Model for processing to output grammatically correct and contextually relevant captions. For proposed model, Standard evaluation metrics used are BLEU, METEOR, GLEU, and ROUGE-L. The experimental results show a great improvement over the present approaches. Obtained results on Flickr8k dataset are BLEU-1 score 0.7024, METEOR score 0.5127, GLEU score 0.2431 and ROUGE-L score 0.5491. It is a model that underscores important progress to be achieved in AI systems for sense-making and describing visual content in the natural language.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ViT-GPT2: Vision Transformer Based Automatic Image Captioning

  • Chanda Srija,
  • B. Padmaja,
  • Yennum Smrithi,
  • Metla Karthikeya

摘要

Automatic image caption generation using deep learning is rapidly bridging computer vision with natural language processing. Envision a system that might be able to carefully study a picture and then describe it in natural language. This technology has huge potential for totally revolutionizing the way interactions with images are undertaken, greatly improving accessibility for visually challenged users, more accurate image search, and streamlined image organization. It makes use of Vision Transformers, especially the ViT-GPT2 model for the extraction of image features from Flickr8k dataset. Features extracted in this stage are further fed into the Vision Encoder Decoder Model for processing to output grammatically correct and contextually relevant captions. For proposed model, Standard evaluation metrics used are BLEU, METEOR, GLEU, and ROUGE-L. The experimental results show a great improvement over the present approaches. Obtained results on Flickr8k dataset are BLEU-1 score 0.7024, METEOR score 0.5127, GLEU score 0.2431 and ROUGE-L score 0.5491. It is a model that underscores important progress to be achieved in AI systems for sense-making and describing visual content in the natural language.