The use of large language models (LLMs) is rapidly increasing. However, in enterprise settings, factuality hallucinations due to a lack of specialized knowledge in LLMs pose a significant issue. To address this, methods that verify text using knowledge graphs (KGs) as external knowledge have been proposed. KGs excel as comparison targets for detecting contradictions since they can accurately describe facts. Developing these methods requires text data containing hallucinations and aligned with KGs. However, existing datasets use hallucinations mainly generated by replacing entities in the text with randomly selected entities from the KGs, which may differ from those actually generated by LLMs and can be easily detectable. Additionally, constructing text corresponding to the facts in a KG is labor-intensive. Therefore, we propose QA2HALL, a framework for generating hallucination detection datasets based on existing knowledge graph question answering (KGQA) datasets. QA2HALL enables semi-automatic generation of text that includes non-trivial contradictions and corresponds to the KGs. This is achieved by utilizing incorrect answers actually generated by LLMs in response to questions from the KGQA datasets. Using this framework, we construct a new hallucination detection dataset and conduct experiments, confirming its practicality and challenge compared to the existing method.