Contrastive learning with the InfoNCE loss is the prevailing method for training dense retrieval models. A key research problem is the adaptation of these models to unseen retrieval corpora through fine-tuning using passages and corresponding synthetically generated queries. While prior work suggests this corpus-specific fine-tuning should improve effectiveness, we uncover a counterintuitive finding: training recent, highly effective embedding models with the conventional InfoNCE loss generally reduces their effectiveness in this setting. We show that this effectiveness degradation is not a trivial issue; it persists despite implementing standard refinements such as passage de-duplication and hard negative mining. We also demonstrate that the limitations of contrastive learning are more fundamental than the well-known problem of false negatives, suggesting that it is insufficient for further fine-tuning already capable models. This failure motivates revisiting cross-encoder listwise distillation, a method that provides richer and more nuanced supervision. We show that distillation yields consistent gains where contrastive learning fails. To enable this investigation, we also analyze the role of the synthetic queries themselves. Our experiments reveal that generating a diverse set of query types (e.g., questions, claims, keywords) significantly enhances corpus-specific adaptation and produces training data that closes the gap in utility to human-written queries. Together, these findings establish that effective corpus-specific fine-tuning requires a combination of robust supervision via listwise distillation and broad data coverage via diverse synthetic queries. Leveraging these insights, we train a BERT-base retriever that achieves effectiveness competitive with the state-of-the-art among comparable models. To support future work, we release our model and code ( https://github.com/manveertamber/cadet-dense-retrieval ).