When a sub-optimal neural network is trying to learn a specific topic to increase performance, there is a tendency to forget what was learnt previously on that topic. This paper presents an adaptive strategy for balancing learning and retention in neural networks, addressing the challenges of catastrophic forgetting and imbalanced learning in image classification tasks. We propose a novel approach that dynamically adjusts class learning priorities based on real-time performance metrics, rather than relying on static weights or oversampling techniques. Our methodology aims to reduce class-level variance, ensuring a fairer distribution of class representation in predictions while accepting slight accuracy trade-offs for dominant classes to enhance minority class performance. We evaluate our approach on three diverse datasets: CIFAR-10, Colorectal Histology, and Imagenette, demonstrating significant improvements in model accuracy and generalization across classes. Our findings indicate that the proposed adaptive framework not only mitigates catastrophic forgetting but also promotes consistent model performance, making it a valuable contribution to the field of deep learning.