The rapid progress of large language models (LLMs) has been applied successfully in service composition and scheduling. However, LLMs will exhibit poor performance when dealing with massive service and complex tasks. To address this challenge, this paper proposes the HMSC-LLMs method, a novel hierarchical multi-agent service composition algorithm based on LLMs, to better interact with users through prompts. Firstly, HMSC-LLMs refers to five roles during service composition, namely, Planner, Manager, Provider, Executor, and Critic. Here each role is based on an LLM agent and responsible for its own domain task. Specifically, the Planner is responsible for decomposing the complex needs into subtasks. The Manager coordinates the activities of various roles and the Provider filters out service. Then the Executor generates service parameters and schedules this service. Furthermore, the Critic supervises the entire service execution workflow to ensure that each role works normally. In our work, the HMSC-LLMs method classified 17000 services and assigned them to multiple agents. Finally, a series of experiments on ToolBench and RapidAPI data sets show that the HMSC-LLMs outperforms traditional single-agent and multi-agent methods in terms of plan accuracy, parameter Accuracy, and hallucination rate.