Traditional decision-making processes often struggle to capture diverse stakeholder perspectives and anticipate potential outcomes. Complex decisions and persuasions might rely on insights and perspectives which might not be available. In this paper, we leverage recent advances in large language models and retrieval-augmented generation to introduce APOLLO—an Architecture and oPen-source system that Orchestrates Large Language mOdels. APOLLO coordinates multiple LLMs by engaging them in collaborative discourse to reach a consensus on user-defined prompts. This system enables HCI and AI researchers and practitioners, and allows them to explore and experiment with LLM-based multi-agents systems in a user-configurable and customisable manner. By providing this flexible platform, APOLLO enables new avenues for studying and designing human-AI interactions, investigating the impact of multi-agent interaction on human behaviour, and ultimately facilitates a deeper understanding of how AI-driven collaboration can enhance human-AI interaction and decision making.