Legal Judgment Prediction (LJP) has become an increasingly significant task in legal AI, primarily focusing on predicting case outcomes based on the factual descriptions of the cases. Collegial panels play a vital role in legal systems worldwide by promoting comprehensive and fair judgments through collective deliberation, mitigating individual biases, and strengthening the legitimacy and authority of decisions. Thus, simulating and applying collegial panel dynamics in LJP is of significant importance. Recent advances in deep learning have led to two main approaches for LJP tasks: large language models (LLMs), which excel at interpreting complex language and providing reasoning, and domain-specific models, which effectively learn task-specific information. In this context, we introduce the Collegial Panel LJP framework (CP-LJP), designed to leverage the advantages of both LLMs and domain-specific models within a panel framework. Specifically, three domain models are utilized to simulate the realistic dynamics of a judicial panel, offering candidate labels by a majority voting mechanism. Subsequently, a large language model synthesizes this information to make the final prediction, articulating the reasoning behind its conclusion. Experiments conducted on real-world datasets demonstrate the effectiveness of the CP-LJP framework, and the results show that our framework achieves overall superior performance compared to state-of-the-art models, with a notable 2.12% improvement in prison term prediction.