In recent years, weakly supervised colonic polyp frame detection based on colonoscopy, widely recognized as the ’gold standard’ for colorectal polyps screening, has attracted significant attention. However, in colonoscopy videos, a minority of normal frames exhibit a distribution that differs from the majority, potentially affecting the assessment of frame abnormality. To address these challenges, we propose TEmory, a model featuring a Temporal Encoder and Memory Unit, designed for weakly supervised colonic polyp frame detection with a comprehensive understanding of normal frame characteristics. Specifically, the Temporal Encoder leverages the contextual information of adjacent frames within video segments, enhancing the encoding’s expressive power. Additionally, the Memory Unit adeptly captures and retains the essential traits of both normal tissues and polyps with heightened precision and exhaustiveness, fortifying the model’s robustness against the nuances of minority normal structures. Experimental outcomes on one of the most extensive and challenging colonoscopy video datasets indicate TEmory’s state-of-the-art performance, showcasing a 1.48% improvement in average precision (AP) over recent advanced techniques. The code of this project is at https://github.com/Liu-Yufei/TEMory .