Deep Learning for medical imaging faces challenges in adapting and generalizing to new contexts. Additionally, it often lacks sufficient labeled data for specific tasks requiring significant annotation effort. This work proposes a novel Continual Active Learning (CAL) framework, called Replay-Based Architecture for Context Adaptation (RBACA), for robust medical image analysis. The Continual Learning (CL) part tackles adaptability and generalizability by enabling lifelong learning from a data stream while mitigating forgetting of previously learned knowledge. On the other hand, the Active Learning (AL) part reduces the number of required annotations for effective training. Based on the automatic recognition of shifts in image characteristics, RBACA employs a CL rehearsal method to continually learn from diverse contexts, and an AL component to select the most informative instances for annotation. We show that RBACA works in domain and class-incremental learning scenarios, by assessing its performance on the segmentation and diagnosis of cardiac images. The results show that RBACA outperforms a baseline framework without CAL, and a state-of-the-art CAL method across various memory sizes and annotation budgets. Our code is available in https://github.com/RuiDaniel/RBACA .