This paper presents a novel image captioning approach that integrates vision transformer (ViT) for feature extraction with long short-term memory (LSTM) networks for caption generation. Leveraging ViT’s self-attention mechanisms, the model effectively captures complex visual dependencies, while the LSTM decoder translates these features into coherent and contextually accurate captions. Using the Flickr8k dataset, the model demonstrates strong performance, achieving high BLEU scores (BLEU-1: 0.9961, BLEU-4: 0.9901), indicating near-perfect alignment with human-annotated captions. This approach surpasses traditional CNN-RNN architectures by generating more fluent and precise descriptions, bridging visual content with natural language. Future improvements may focus on enhancing caption diversity and exploring additional evaluation metrics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ViT-LSTM: Image Caption Generation

  • Astha Kathar,
  • Mithil Gogri,
  • Arsh Mathur

摘要

This paper presents a novel image captioning approach that integrates vision transformer (ViT) for feature extraction with long short-term memory (LSTM) networks for caption generation. Leveraging ViT’s self-attention mechanisms, the model effectively captures complex visual dependencies, while the LSTM decoder translates these features into coherent and contextually accurate captions. Using the Flickr8k dataset, the model demonstrates strong performance, achieving high BLEU scores (BLEU-1: 0.9961, BLEU-4: 0.9901), indicating near-perfect alignment with human-annotated captions. This approach surpasses traditional CNN-RNN architectures by generating more fluent and precise descriptions, bridging visual content with natural language. Future improvements may focus on enhancing caption diversity and exploring additional evaluation metrics.