Detecting the prediction errors of machine learning model is critical for preventing undesirable consequence and enhancing the safety of machine learning systems. However, existing error detection methods often operate as black-box models, making their outputs difficult to interpret and consequently hindering user trust. Additionally, with the enhancement of model accuracy, the occurrences of errors decrease significantly, posing challenges in distinguishing error samples due to the imbalance between correct and erroneous predictions. This paper introduces a novel error detection method called ImExED, which is explainable and accounts for the imbalance between correct and erroneous samples. By integrating Explainable Boosting Machine (EBM) with techniques designed for handling class imbalances, ImExED improves both the interpretability and effectiveness of error detection. Through evaluations on diverse datasets and model families, along with an ablation study on class imbalance techniques, ImExED outperforms current methods in error detection while providing insights into the correctness or incorrectness of model predictions.