Deep learning methods have demonstrated promising results in cervical lesion cell detection. Training detection models that generalize well typically require a large amount of cell-level annotations that are expensive and time-consuming to obtain. Instead, weak slide-level annotations, which entail assigning a gigapixel whole slide image (WSI) with a single label, are easier to acquire. However, due to significant differences in annotation scales, they cannot be directly utilized to assist in the training of cervical cell detectors. To address this challenge, we propose a Twin-memory augmented Multiple Instance Learning (Twin-MIL) framework to refine cervical lesion cell detection. Firstly, we utilize the multiple instance learning to bridge the gap between cell-level and slide-level tasks. Then, we reduce false positives in conventional MIL by introducing a twin-memory module, which improves the classification capability by capturing more discriminative patterns of positive and negative cells. We also propose uncertainty-regulated negative instance learning to enhance the robustness of negative latent space against noisy instances and its separability from the positive one. Experiments indicate that our method is effective in enhancing different detection models trained on the datasets with varying annotation levels.