Machine learning-based access control (MLBAC) shows promise in effectively determining access in complex scenarios where a trained ML model makes decisions. When access policies change, the underlying ML model must be updated to accommodate these changes. This process requires a portion of past training data, known as Replay Data, along with the new changes to retain existing knowledge and avoid challenges like catastrophic forgetting. Traditionally, Replay Data is selected randomly, constituting a large portion (approximately 25%) of the historical data, and often does not adequately represent the overall data distribution. This paper proposes a systematic approach to selecting Replay Data using active learning-based data distillation, resulting in a more compact (approximately 10% of the training data) and representative dataset. We thoroughly examine the proposed method and assess the effectiveness of the developed Replay Data. Our evaluation demonstrates that, even with this more compact Replay Data, the ML model can retain its past knowledge with impressive accuracy, ranging from 96.5% to 98.5%.