DR.SIMON: Domain-Wise Rewrite for Segment-Informed Medical Oversight Network
摘要
Humans are capable of understanding language, even when encountering unfamiliar words. Rather than requiring precise definitions, we often infer meaning from the surrounding linguistic context or visual cues. Inspired by this capability, we address the long-standing challenge of aligning medical terminology in queries with visual content for temporal grounding in medical videos. While bridging this gap typically relies on costly, domain-specific fine-tuning, such methods frequently lack generalization and struggle to adapt to newly coined or rarely encountered terms. To deal with this limitation, we present DR.SIMON (Domain-wise Rewrite for Segment-Informed Medical Oversight Network), a simple yet efficient query-rewriting framework that runs on a frozen backbone. DR.SIMON first segments the video into coarse events, then rewrites the user query into visually explicit paraphrases under global visual context, and finally localizes the most relevant segment. Evaluated on MedVidCL, DR.SIMON achieves remarkable gains over recent video-LLMs—without any additional training. Our results show that mitigating lexical misalignment alone can unlock substantial performance improvements and provide a scalable route to keep pace with continually emerging medical vocabulary. Code will be released at https://drsimon-rewrite.github.io/ .