This paper studies the problem of evaluating predictive maintenance models in the presence of reflexivity in a case study of pharmaceutical manufacturing. Reflexivity occurs when models influence the outcome they are designed to predict, creating feedback loops that affect future data distributions. In pharmaceutical manufacturing, these feedback loops arise when predictive models flag potential failures, triggering maintenance interventions that prevent those failures from occurring. While these interventions are generally beneficial, they create challenges for model evaluation and retraining. In a case study of a pill manufacturing system, we simulate and analyze how the predictions of a classifier affect the data distribution. We show that positive predictions systematically reduce the frequency of anomalous events and eventually compromise the performance of the classifier as the data distribution shifts away the one observed in the original training data. Overall, our study provides insights about model monitoring, retraining strategies, and performance evaluation in real–world industrial settings where model predictions influence system’s behavior.