Large language models (LLMs) with in-context learning (ICL) have notably boosted the performance in text-to-SQL, with prior efforts concentrating on exclusive SQL prompts to enhance reasoning ability. However, it is still challenging to further enhance the operational efficiency and inference performance of LLMs. To tackle this challenge, we propose MR-SQL, a multi-level retrieval-based LLM framework consists of three specially designed retrievers. The retrievers collaborate to retrieve valuable information for the target question, which not only reduce schema size and minimize the interference noise, but also enhance the reasoning capability of LLMs through more similar Chains of Thought (CoT). Concretely, Table-Retriever and Column-Retriever retrieve concise tables and columns from original large databases with redundant schema information. Example-Retriever select similar few-shot examples for more targeted CoT. Experiment results indicate that MR-SQL increases the execution accuracy on the BIRD and Spider validation sets by +2.54% and +1.15% respectively.