One of the major obstacles to building a domain specific Question Answer (QA) system is the lack of domain specific training data to fine-tune a pre-trained Large Language Model (LLM). Plain domain specific corpus is often abundant, but it is too expensive to convert it into QA pairs for fine-tuning. This research introduces a LLM-based integrated REtrieval and GENeration model (REGEN) to extract QA pairs from plain texts, such as text books, technical documentations and work logs. The model was fine-tuned on few-shot labeled QA pairs. It was further applied to plain texts, extracting abundant QA pairs to fine-tune another LLM to build a domain specific QA system. We demonstrated that our model performs significantly better than the baseline model in domain specific tasks with only 1k manually labeled QA pairs and 10M token extracted from international freight domain specific corpus. In the performance test, our model obtained about 70 points in the International Freight Forwarding Qualification Examination (IFFQE), 30 points more than the baseline model. This research presents an ideal choice for adopting LLM as a QA system into new domains, especially when there is little domain QA data available.