<p>Large language and vision models have demonstrated remarkable performance, but their high pre-training and fine-tuning costs have led to growing interest in accelerating training through dataset pruning. Traditional pruning approaches typically rely on sample loss to identify and retain the most "difficult" samples. However, as the pruning rate increases, the training frequency of each sample becomes more uniform, which can result in the underfitting of critical or general samples. We identify this phenomenon as Low-Frequency Learning (LFL), where the model fails to retain knowledge of the majority of samples. In this work, we decompose the scoring function of LFL, providing a theoretical analysis of its inefficiencies. To counteract this issue, we propose a novel enhancement to the scoring function by introducing a memory term designed to strengthen the model’s ability to memorize essential data. We also offer an approximation for this memory term. Additionally, we extend this concept to Self-Supervised Learning (SSL), marking the first investigation into the role of memory in SSL. By leveraging contrastive learning, we derive the memory term both theoretically and experimentally. Based on these insights, we introduce Enhance Memory Pruning (EMP), a technique that mitigates memory loss under high pruning rates by improving the model’s data retention. EMP has been evaluated across various tasks, including image classification, natural language understanding, and model pre-training. Our experiments demonstrate that EMP significantly enhances model performance, particularly under extreme pruning conditions. For instance, in the CIFAR100-ResNet50 pre-training task with a 70% pruning rate, EMP outperforms current state-of-the-art methods by 2.2.</p>