<p>While generative AI is increasingly used to extract structured knowledge from text, human input and painstaking efforts are still needed to monitor and fix AI hallucinations. This paper investigates how to support human-AI collaboration in the remedy of hallucinations during the transformation of text into structured knowledge entities. We propose KGenAI, a framework that supports a two-way human-AI feedback loop. It aims at keeping the human actively involved in detecting and addressing AI hallucinations. The framework guides humans to instruct AI how to explain itself, and leverages AI to use its explanations as hallucination signals for humans to further finetune the model through targeted training. We validate KGenAI through a use case in transforming product reviews into knowledge graph entities that support human-centered recommender systems. Experiments using the LLaMA2-7B model on an NVIDIA A100 GPU cluster show that the baseline (non-finetuned) model achieved an overall macro F1 score of 16.0% with a hallucination rate of 61.1%. After finetuning the model following our human-AI feedback framework, entity detection performance substantially improved. Initial results of two rounds of finetuning raised F1 scores to 67.07%, reduced hallucination rates to 11.69%, and decreased human review time on average from 10&#xa0;min to 0.21&#xa0;min per review. These results provide initial evidence of the framework’s effectiveness in reducing hallucinations and optimizing human effort while improving AI output quality. This work contributes to enhancing human-AI collaboration for transforming text to knowledge through human-guided AI explanations.</p>