Accurately segmenting biomarkers in fundus images is crucial for the recognition of retinal diseases. While most existing segmentation methods are fully supervised and limited to handling single tasks, fundus image datasets labelled with partial classes are sequentially constructed from various medical institutions in real-world scenarios. Consequently, dynamically extending a model to new datasets and classes is essential for training a unified segmentation model. It is rather challenging without the availability of previous datasets and annotations due to storage and privacy restrictions. In this paper, we propose a novel replay-free continual segmentation method for fundus images. To address the issue of overlapping class regions in fundus images, we introduce a learning procedure that generates pseudo labels separately for old classes. Additionally, to tackle the problem of imbalanced class distribution in fundus images, we combine class-agnostic knowledge distillation and class prototype contrastive learning for old knowledge transfer to achieve a better balance between stability and plasticity. Furthermore, considering the impact of different task orders on the quality of generated pseudo labels, we employ a dynamic forget gate to selectively utilize knowledge distillation for further improving plasticity. Through extensive experiments conducted on multiple fundus image datasets with varying task orders, our proposed method demonstrates its effectiveness by significantly outperforming other comparison methods at every incremental step.