We present a rigorous theoretical foundation for multi-agent large language model (LLM) systems. Our framework, Control-Theoretic Multi-Agent LLM Dynamics (CT-MALD), models each agent as a controlled jump-diffusion with actuation delay and committed (non-preemptive) action windows. We prove a dynamic programming principle (DPP) and a viscosity characterization for the resulting nonlocal Hamilton–Jacobi–Bellman (HJB) equation with delay and commitment. Horizontal collaboration is modeled via f-divergence-bounded information exchange; we prove exponential reductions in expected time-to-decision under Rényi divergence budgets. Vertical coordination is formulated as a continuous-time hierarchical Stackelberg game; we prove equilibrium existence, uniqueness under strict quasi-concavity, and sensitivity via the implicit function theorem. For prompt/policy selection we develop Thompson-Regularized Gaussian Process Contextual Bandits (TR-GPCB) and prove high-probability regret bounds with delayed, heteroscedastic feedback and instance-dependent refinements. Finally, we propose a Hierarchical Empirical Bayes Correction (HEBC) mechanism and prove conjugate posteriors, posterior contraction, an optimal decision threshold, and strict average error reduction.