Knowledge Graph Question Answering (KGQA) has long been an emerging question in the NLP community. However, KGQA systems may face the issues of severe data imbalance and limited data amount. To solve them, our study improves KGQA by integrating natural language prompts and enhancing text-graph integration. Specifically, we use LLMs to automatically compile prompts to augment the existing limited data without any human labor of curating more data. On a Wikidata dataset, extensive experiments show that our collected prompts can lead to an excellent F1 score of 0.9424, surpassing the baselines by 11.08%. Then, we also study instruction-tuned LLMs on KGQA and compare them with our fine-tuned smaller models. Then, we also conduct various ablation studies to show the effectiveness of our method.