Knowledge-Enhanced Hyperbolic Language-Image Pretraining for Zero-Shot Learning
摘要
In light of the inherent entailment relations between images and text, hyperbolic point vector embeddings, which leverage the hierarchical structure of hyperbolic space, have been used for visual-semantic representation, showing significant advantages in zero-shot learning tasks. However, unlike general image-text alignment tasks, the availability of high-quality paired image-text data in the medical domain is limited. This scarcity presents challenges for visual language models, hindering their ability to effectively comprehend free-text medical reports and the associated images. Moreover, many medical terms are complex, specialized, and abstract, and embeddings derived solely from raw imaging report texts often fail to generalize effectively. To address these challenges, we propose MkCLIPH, a hyperbolic space image-text alignment pre-training method that incorporates medical domain knowledge. MkCLIPH models the visual-semantic hierarchical partial order relationship through hierarchical entailment angle modeling and integrates medical domain knowledge as a prior to enhance the representation of medical image-text data. This improves generalization and interpretability. Experimental results demonstrate that our method outperforms baseline approaches in terms of interpretability and performance across a range of zero-shot tasks and datasets.