This article proposes an adaptive learning-based optimal cooperative control method for nonlinear multi-agent systems (MASs) subject to output constraints. This constraint is more comprehensive, as the constrained boundaries are not only time-varying and asymmetric but also positive and negative switching. To address asymmetric time-varying output constraints, a system transformation method is designed to obtain equivalent unconstrained MASs utilizing a one-to-one state mapping. Subsequently, a neural network (NN)-based distributed observer is developed to simplify the cooperative control problem into multiple trajectory tracking problems. The proposed distributed observer overcomes the limitation of prior knowledge of leader dynamics, and ensures that the observer errors can converge in a finite time. For optimizing system performance, an adaptive dynamic programming algorithm is developed. Leveraging the powerful learning ability of the critic NNs, the approximate optimal controller for each agent can be obtained. Lyapunov analysis provides a rigorous proof of the boundedness of all signals in the closed-loop systems. The efficacy of the proposed approach can be substantiated through simulation results conducted on multiple single-link robot arm systems.