False negatives, where semantically relevant image-text pairs are incorrectly labeled as unmatched, present a significant challenge in image-text retrieval (ITR). However, existing approaches addressing this issue face two major limitations. First, the generated soft labels are often of poor quality. Second, reranking methods typically rely on access to the full set of queries and targets, which limits their applicability in various scenarios. To tackle the limitations of existing approaches, we propose Text Soft label and KL similarity (TSKL) method consists of two independent plug-in modules. In the training phase, the Text Soft Label (TSL) module generates soft similarity labels from image captions. In the testing phase, the KL similarity (KLsim) module. KLsim only requires the current query and all targets to compute similarity scores. Both plug-in modules can be easily integrated into existing ITR models. We evaluate TSKL on three widely used datasets, Flickr30K, MSCOCO, and CUB Captions. Experimental results show our method consistently improves the performance of base models by an average of 12.3% in terms of Rsum, and effectively mitigates the impact of false negatives by providing more accurate similarity rankings.