Scholarlink

<p>Multimodal tagging is essential for content understanding by assigning concise, semantically relevant tags to visual inputs. However, real-world tagging is inherently open-ended: user-generated content is noisy, long-tailed, and continuously evolving, challenging conventional closed-set or open-vocabulary classification methods. We propose <i>Open-Tag</i>, a generative framework for <i>Open-world Multimodal Tagging</i> that produces unordered, variable-length tag sequences in natural language without relying on predefined tag sets. Open-Tag introduces two key innovations: (1) an <i>Order-Prompted Tag Sequence Generation</i> that maps learnable, order-agnostic queries to latent tag semantics, enabling permutation-invariant tag generation, and (2) a <i>Multi-Source Retrieval-Augmented Generation</i> that fuses tag candidates from heterogeneous retrieval systems across visual, textual, and metadata modalities. A score normalization and aggregation strategy ensures robust fusion, enhancing the diversity and grounding of generated tags. To evaluate Open-Tag, we construct two large-scale datasets: <i>CREATE-Tag</i> (Chinese video) and <i>PEXEL-Tag</i> (English image), with over 3M videos and 160K images with tens of thousands of real-user tags. We propose a novel open-set evaluation metric, <i>Tag Gain</i>, to quantify the generation of relevant but previously unseen tags. Experiments show that Open-Tag outperforms state-of-the-art baselines on closed-set F1 and open-set Tag Gain, highlighting its generalization and novel tag discovery capabilities. Project webpage: <a href="https://createbenchmark.github.io/open-tag/">https://createbenchmark.github.io/open-tag/</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Open-Tag: A Generative Framework for Open-World Multimodal Tagging

  • 期刊论文

  • 发表日期：

    2026年5月21日

查看全文
  • Ziqi Zhang,
  • Zongyang Ma,
  • Peijin Wang,
  • Bing Li,
  • Chunfeng Yuan,
  • Weiming Hu

摘要

Multimodal tagging is essential for content understanding by assigning concise, semantically relevant tags to visual inputs. However, real-world tagging is inherently open-ended: user-generated content is noisy, long-tailed, and continuously evolving, challenging conventional closed-set or open-vocabulary classification methods. We propose Open-Tag, a generative framework for Open-world Multimodal Tagging that produces unordered, variable-length tag sequences in natural language without relying on predefined tag sets. Open-Tag introduces two key innovations: (1) an Order-Prompted Tag Sequence Generation that maps learnable, order-agnostic queries to latent tag semantics, enabling permutation-invariant tag generation, and (2) a Multi-Source Retrieval-Augmented Generation that fuses tag candidates from heterogeneous retrieval systems across visual, textual, and metadata modalities. A score normalization and aggregation strategy ensures robust fusion, enhancing the diversity and grounding of generated tags. To evaluate Open-Tag, we construct two large-scale datasets: CREATE-Tag (Chinese video) and PEXEL-Tag (English image), with over 3M videos and 160K images with tens of thousands of real-user tags. We propose a novel open-set evaluation metric, Tag Gain, to quantify the generation of relevant but previously unseen tags. Experiments show that Open-Tag outperforms state-of-the-art baselines on closed-set F1 and open-set Tag Gain, highlighting its generalization and novel tag discovery capabilities. Project webpage: https://createbenchmark.github.io/open-tag/.