The effectiveness of self-supervised learning, particularly in vision–language models like CLIP, is largely influenced by the quality and balance of the data rather than the model architecture. Internet-crawled datasets often contain noise and exhibit long-tail distributions, highlighting the need for more robust curation strategies. Existing filtering methods that leverage specificity or quality metrics (e.g., HYPE, DFNs) can remove meaningless or low-quality image–text pairs, yet they often overlook the distribution bias present in clustered sample groups. In this work, we introduce a simple yet effective dataset curation framework that employs a k-means-based method to identify and eliminate noisy and redundant samples in CLIP-based datasets by sampling in a balanced way from refined clusters. Our approach is motivated by the initial observation that both image-only and text-only crawled datasets tend to manifest long-tail distributions, potentially hindering downstream performance. Comprehensive experiments on image, text, and multimodal data demonstrate that our method outperforms baseline CLIP score filtering (which retains pairs with cosine similarity scores above a defined threshold) and competes favorably against alternative strategies on the DataComp benchmark when the text modality is balanced.