<p>Single-cell RNA sequencing (scRNA-seq) plays a vital role in studying cellular heterogeneity and gene expression patterns. However, the sequencing dropout phenomena still pose a significant challenge. Genes with low expression levels may be misidentified as exhibiting zero expression owing to limitations in sequencing depth and technical noise. This results in increased data sparsity and compromises the accuracy of subsequent analyses. Thus, a novel method, MoDET (Dual-level Momentum Distillation Method with Extreme Thresholding), has been proposed. MoDET employs a label-guided model and an extreme threshold mechanism to enhance cellular representation learning. Experiments demonstrate that MoDET significantly improves clustering performance of the gene expression matrix, with enhancements ranging from 3% to 20% across seven real-world datasets. Cross-batch training and evaluation experiments demonstrated that MoDET effectively mitigates batch effects, achieving an average performance improvement of 5%-7%. Concurrently, it exhibits superior accuracy in identifying rare cell types, outperforming other methods by 3%-20%. Ablation studies confirm that the dual-level momentum distillation boosts performance by 4%–20%, and the extreme threshold mechanism adds an additional 2%–15% improvement. Interpretability analysis shows that the extreme threshold makes the model’s decision-making process more transparent. Moreover, MoDET surpasses methods incorporating advanced modules, thereby demonstrating its efficacy in addressing the sparsity challenges inherent in scRNA-seq datasets. The compiled source codes are accessible at https://github.com/gladex/MoDET.</p> Graphical Abstract <p></p>