Image-text retrieval aims to establish robust semantic alignment between visual and textual representations, facilitating precise bidirectional matching across images and texts. However, existing image-text retrieval datasets often assume a one-to-one mapping between images and captions, overlooking their inherent many-to-many relationships in real-world scenarios. This oversight results in an average of 89 false-negative captions per image in the MSCOCO dataset, with only one pairing considered matched, while also causing 178 incorrectly judged images per caption. These instances, which we term cross-matching pairs, inevitably compromise retrieval performance when utilized in training. In this paper, we propose M \(^3\) ITR, a simple but effective pipeline, which models the many-to-many relationships between images and texts for image-text retrieval to address this limitation, enhancing the robustness and accuracy of image-text retrieval. Specifically, we start by detecting cross-matching pairs within datasets through a two-step verification pipeline powered by Qwen-VL and GPT-4. During training, we propose the Universal Variable Learning Rate (UVLR) strategy, enabling models to robustly learn from detected cross-matching pairs through image-text matching loss. Finally, to ensure a more accurate and reliable evaluation of the model’s image-text retrieval capabilities, we manually revised the cross-matching pairs in the test set. We applied the UVLR strategy to three widely-used image-text retrieval backbones. Extensive evaluations on two benchmarks demonstrate that UVLR delivers substantial performance improvements, achieving gains of up to 7.0% on unmodified test sets and 5.8% on our revised test sets, respectively.