The quality of Machine Learning (ML) models strongly depends on the quality of the input data, as such Feature Engineering (FE) is often required in ML. In addition, with the proliferation of ML-powered systems, especially in critical contexts, the need for interpretability and explainability becomes increasingly important. Since manual FE is time-consuming and requires case specific knowledge, we propose KRAFT, an automated FE (AutoFE) framework that leverages a Knowledge Graph (KG) to guide the generation of interpretable features. Our hybrid AI approach combines a neural generator to transform raw features through a series of transformations and a knowledge-based reasoner to evaluate features interpretability using Description Logics. The generator is trained through Deep Reinforcement Learning to maximize the prediction accuracy and the interpretability of the generated features. Extensive experiments on real datasets demonstrate that KRAFT significantly improves accuracy while ensuring a high level of interpretability. This work highlights the potential of integrating semantic web and reasoning in FE, paving the way for interpretable ML models.