We introduce a novel method MedRIS for the challenging task of Medical Referring Image Segmentation (Medical RIS): segment the lesions described in a medical report. MedRIS addresses two major challenges: (I) a single text description may correspond to multiple lesions (one-to-many reference); (II) annotation may be uncertain due to complex appearance of lesions and subjectivity of annotators. To solve (I), mask self-augmentation separates the original mask into independent or combinations of lesions. To solve (II), ChatGPT and organ position matching are leveraged to reformulate textual descriptions into content-diverse sentences that are consistent with the augmented masks. Organ-related positional attention combined with real-world medical knowledge focuses the model on critical image areas. Exponential moving average (EMA) and local annotation correction are used to suppress the effect of noisy annotations during training. Extensive experiments on public and in house chest X-ray datasets show that MedRIS outperforms the state-of-the-art methods by a large margin.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Medical Referring Image Segmentation: Addressing Multi-Lesion Reference and Annotation Uncertainty via Vision-Language Fusion

  • Yuanyang He,
  • Chun-Mei Feng,
  • Yang Zhou,
  • Lionel Cheng,
  • Anh Tran,
  • Gideon Ooi,
  • Choon Thng,
  • Salman Khan,
  • Wangmeng Zuo,
  • Yong Liu,
  • Juergen Schmidhuber

摘要

We introduce a novel method MedRIS for the challenging task of Medical Referring Image Segmentation (Medical RIS): segment the lesions described in a medical report. MedRIS addresses two major challenges: (I) a single text description may correspond to multiple lesions (one-to-many reference); (II) annotation may be uncertain due to complex appearance of lesions and subjectivity of annotators. To solve (I), mask self-augmentation separates the original mask into independent or combinations of lesions. To solve (II), ChatGPT and organ position matching are leveraged to reformulate textual descriptions into content-diverse sentences that are consistent with the augmented masks. Organ-related positional attention combined with real-world medical knowledge focuses the model on critical image areas. Exponential moving average (EMA) and local annotation correction are used to suppress the effect of noisy annotations during training. Extensive experiments on public and in house chest X-ray datasets show that MedRIS outperforms the state-of-the-art methods by a large margin.