Comics represent the complex way humans can communicate and expose ideas. We investigate how a deep learning pipeline performs in creating a narrative that describes the story occurring at comics sequences. The framework relies on the multimodal BLIP-2 architecture to bridge the gap between the sequence of images and the text description. Our analysis unveils the relevance of prompt engineering, both in the definition of the most effective prompting, as well as in the strategy adopted for the ingestion of visual information. In this case, it seems preferable to present images sequentially, possibly enriched with some context from previous steps, and complete the process with a final summarization step. The results are promising, revealing that in many situations the pipeline is able to generate text narratives that are close to the semantic space of the real descriptions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Capturing the Narrative: A Deep Learning Pipeline for Comics Sequences

  • Goncalo Marouvo,
  • Francisco Pereira

摘要

Comics represent the complex way humans can communicate and expose ideas. We investigate how a deep learning pipeline performs in creating a narrative that describes the story occurring at comics sequences. The framework relies on the multimodal BLIP-2 architecture to bridge the gap between the sequence of images and the text description. Our analysis unveils the relevance of prompt engineering, both in the definition of the most effective prompting, as well as in the strategy adopted for the ingestion of visual information. In this case, it seems preferable to present images sequentially, possibly enriched with some context from previous steps, and complete the process with a final summarization step. The results are promising, revealing that in many situations the pipeline is able to generate text narratives that are close to the semantic space of the real descriptions.