Scholarlink

<p>When describing pictures from the point of view of human observers, the tendency is to prioritize eye-catching objects, link them to corresponding labels, and then integrate the results with background information (i.e., nearby objects or locations) to provide context. Most caption generation schemes consider the visual information of objects, while ignoring the corresponding labels, the setting, and/or the spatial relationship between the object and setting. This fails to exploit most of the useful information that the image might otherwise provide. In the current study, we develop a model using object’s tags to supplement insufficient information of visual object features as well as establishing relationship between objects and background features based on relative and absolute coordinate information. In addition, we propose an attention architecture to account for all of the features in generating an image description. The effectiveness of the proposed geometrically aware dual transformer encoding visual and textual features (GDVT) is demonstrated in experiment settings with and without pre-training. The effectiveness of our added features is verified in the case studies.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GDVT: geometrically aware dual transformer encoding visual and textual features for image captioning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月16日

查看全文
  • Yu-Ling Chang,
  • Hao-Shang Ma,
  • Shiou-Chi Li,
  • Bijay Prasad Jaysawal,
  • Jen-Wei Huang

摘要

When describing pictures from the point of view of human observers, the tendency is to prioritize eye-catching objects, link them to corresponding labels, and then integrate the results with background information (i.e., nearby objects or locations) to provide context. Most caption generation schemes consider the visual information of objects, while ignoring the corresponding labels, the setting, and/or the spatial relationship between the object and setting. This fails to exploit most of the useful information that the image might otherwise provide. In the current study, we develop a model using object’s tags to supplement insufficient information of visual object features as well as establishing relationship between objects and background features based on relative and absolute coordinate information. In addition, we propose an attention architecture to account for all of the features in generating an image description. The effectiveness of the proposed geometrically aware dual transformer encoding visual and textual features (GDVT) is demonstrated in experiment settings with and without pre-training. The effectiveness of our added features is verified in the case studies.