In real-world environments, text classification faces limitations due to finite data and the closed-world assumptions combined with distributional identity in machine learning models. Leading to data with a distribution shifted from the learned one (in-distribution, ID), called out-of-distribution (OOD) data. OOD data present challenges related to safety, robustness, and the model’s ability to discover new patterns. The literature on OOD detection mainly focuses on safety and robustness, leaving aside the discovery area where we can leverage the distinction between valuable OOD data, such as new classes, and anomalies, like outliers. To address this, we propose fine-tuning pre-training models with labeled OOD data and a regularization term for the outlier exposure method with samples of a mixture of beta distributions, forming a distribution with two peaks in the probability interval. This approach enhances the differentiation between two types of OOD data, near-OOD and far-OOD. Our experiments use RoBERTa, a BERT-based model, fine-tuned with LoRA, where the results show improved differentiation between near-OOD and far-OOD data without compromising performance on the principal task of multi-class classification and the base detection of ID and OOD. Using AUROC and FPR@95 metrics, our results indicate that this modified OOD detection with a beta approach for outlier exposure (OE) maintains good robustness and enhances the discovery capabilities of text classification models in real-world applications.