Gastroscopic Lesion Detection (GLD) is one of the critical tasks within computer-assisted gastroscopic diagnostics. Endoscopists adopt a pattern-based philosophy for GLD: they identify and summarize typical sub-category patterns with specific medical meanings and conduct GLD based on these patterns. However, the current gastroscopic lesion detectors follow the classical data-driven deep-learning-based training paradigm, which differs from the endoscopists’ diagnosis process and leads to low interpretability, limiting their performance and potential for daily clinical practice and patient care. The intuitive data-driven solution with sub-category pattern labels may work but it requires expensive annotation costs. In this work, we imitate the pattern-based philosophy with limited labels and propose a Pattern-Anchored Adaptive Prototype Learning (PAAPL) for Gastroscopic Lesion Detection. PAAPL consists of a Prototype-based Gastroscopic Lesion Detector (PGLD) and a Pattern-Anchored Adaptive Learning (PAAL) strategy. PGLD achieves sub-category pattern detection based on similarity to prototypes. PAAL proposes a vector-wise prototype formulation and an adaptive prototype update strategy to anchor prototypes to limited-annotated patterns with specific medical meanings and adaptively learn pattern characteristics from unannotated data in GLD datasets. We evaluate PAAPL on the LGLDD and Endo21 datasets, demonstrating its ability to learn and detect sub-category patterns trained with limited annotations. By doing this, PAAPL enhances detector interpretability and yields significant performance improvement (+3.7AP on LGLDD/+5.4AP on Endo21).