The process of evaluating machine learning models is essential for determining their performance and generalization capabilities beyond training data. Model evaluation assesses how well a model can predict unseen data, ensuring it meets the requirements of the problem at hand. This evaluation involves understanding concepts like overfitting and underfitting, where models either excessively fit to training data or fail to capture underlying patterns, respectively. To measure model performance, various metrics are employed depending on the problem type. For regression models, metrics like Mean Absolute Error, Mean-Squared Error, and R-squared are commonly used. In classification models, metrics such as accuracy, precision, recall, specificity, and F1 score are employed, along with techniques like ROC curves and AUC. Validation techniques like k-fold cross-validation help mitigate biases and variance in performance estimation. Careful consideration of these evaluation methods ensures the development of robust machine learning solutions suitable for specific problems. Evaluating machine learning models is a crucial step during data analysis (Ding et al., IEEE Signal Processing Magazine 35(6):16–34, 2018). Model evaluation allows to assess whether a chosen model performs well not just on the training data, but also on new, unseen data and is thus able to generalize beyond the given dataset. Through model evaluation the data scientist can assess to what extent the chosen model not only learns the features of the training data, but more importantly, how well the model is able to make predictions on data it has never seen before, so-called potential observables. The scientist can decide whether the developed model meets the requirements of the specific problem under investigation.