Feature selection is crucial in building Machine Learning (ML) models. A model trained on selected features can outperform models trained on all available features in predictive accuracy. Most feature selection algorithms focus on the predictive accuracy of the models. Hence, feature selection algorithms incorporate various statistical methodologies to maximize predictive accuracy. While this serves the purpose of maximizing accuracy, consistency is needed to explain the model’s decision. A regression model can be useful for eXplainable Artificial Intelligence (XAI) if and only if the coefficients show consistent signs (positive or negative) despite the inherent variability in data. This work demonstrates that linear regression models built using features selected by traditional approaches exhibit poor consistency of coefficient signs. This inconsistency in the sign of coefficients can hinder the understanding of feature influence on the target. To address this, we propose a novel feature selection algorithm that selects only those features that minimize the fluctuation of the model’s coefficients’ sign, i.e., consistent features. Our experimental results on three different public datasets and two regression techniques demonstrate the effectiveness of our approach. Thus, models built on the selected features using our approach exhibit better consistency of the coefficients’ sign than models built on features selected using traditional, with minimal impact on predictive accuracy. This substantial improvement in consistency shows that our approach cannot only compete with existing approaches in terms of accuracy but also outperform them in terms of consistency, making it a valuable tool for XAI applications.