Large language models (LLMs) are able to answer reasoning questions through generating reasoning processes, such as Chains of Thoughts (CoT) and Trees of Thoughts (ToT). To retrieve external documents effectively for knowledge-intensive multi-step question answering (QA), we propose Retrieval Augmented Tree of Thoughts (RAToT), a retrieval framework which performs retrieval and locally optimal reasoning iteratively. RAToT inspires LLM to explore diverse reasoning paths via separating retrieved documents and selects the most promising one for each step. RAToT, which uses gpt-3.5-turbo-instruct as backbone LLM, retrieves documents with less noise than IRCoT on three open-domain multi-step QA datasets, HotpotQA, 2WikiMultihopQA and MuSiQue, and further enhances the performance of LLM QA on MuSiQue. RAToT can retrieve documents wisely and flexibly to mitigate the hallucinations in open-domain multi-step QA tasks (Code and data are available at https://github.com/XMLAleph/RAToT ).