<p>Traditionally, the designs of control policies for the group formation control (GFC) problem of multi-agent systems (MASs) rely on accurate system dynamics, which have been proven challenging to obtain in the complex real world, leading to unknown MASs. In this respect, this paper investigates the distributed GFC problem of heterogeneous nonlinear MASs with unknown dynamics. First, an effective and more flexible communication topology is designed to achieve communication configuration among agents. Then, the GFC problem is formulated by defining the local group neighborhood formation error and performance index for each agent under the designed communication topology. By developing an effective policy iteration algorithm and establishing a corresponding actor-critic neural network framework, a novel model-free GFC algorithm is proposed, overcoming the reliance on system dynamics for control design. The proposed model-free algorithm can seek the optimal GFC control policy online using system operation data to achieve GFC and minimize control cost, thus yielding better control performance and demonstrating superior practicality in practical applications compared with traditional offline model-based methods. Finally, two simulation examples demonstrate the effectiveness and superiority of the developed model-free algorithm for solving the GFC problem.</p>