Task-oriented dialog (TOD) systems use external knowledge sources to help users accomplish specific tasks. While most current TOD research focuses on simple information-collecting tasks in a slot-filling framework, multi-step reasoning tasks like troubleshooting remain underexplored. Leveraging the advancements of large language models (LLMs), we propose a novel LLM-based multi-agent learning framework to build troubleshooting dialogue systems and evaluate the effectiveness of various multi-agent learning settings in a TOD system. Our results show that LLMs designed for open-domain dialog face challenges when directly applied to TOD systems, but with multi-agent cooperative enhancements, LLMs can achieve commendable performance.