The large number of agents and complex interactions lead the control and performance optimization of massive multi-agent systems challenging. In this paper, we investigate an online federated adaptive critic-density learning-based control method to solve the massive multi-agent tracking control problem. First, inspired by the mean-field game, each agent regards the complex interaction with all other agents as an average or collective influence. Then, a novel cost function incorporating a mean-field interaction is designed without requiring to describe the complex interactions between agents and the huge burden of communication resources among the large number of agents. Furthermore, the Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck (FP) equations are constructed to derive the optimal control policies and the probability density function. To solve the HJB and FP equations, a novel online federated adaptive critic-density learning algorithm is proposed, which can avoid excessive differences in control strategies among agents and satisfy the required convergence condition for the MFG. Finally, a simulation example is provided to show the effectiveness of the present control scheme.