This paper conducts an in-depth study on the task of unmanned surface vehicle (USV) swarm encirclement in complex maritime environments by employing a novel network-based model known as the game theoretic utility tree (GUT). Specifically, the GUT model, which can handle the potential relationships between multiple agents and provides a utility maximization decision framework for each agent, is first utilized. In the field of pursuit-evasion games, it offers a cooperative pursuit strategy for capturing evaders, where agents in the multiagent system (MAS) can adopt appropriate decision-making strategies on the basis of their respective tasks and objectives. In addition, how to achieve effective collaborative operations of the USV swarm in complex maritime environments to complete the encirclement task is further investigated. The effectiveness of the proposed methods is verified through simulation experiments, which demonstrate that when factors such as the number of encirclement vehicles and obstacles are considered, the GUT model significantly improves the efficiency and success rate of encirclement. These studies, address how to guide the USV swarm to carry out effective encirclement in complex maritime environments via the game theoretic utility tree model, thereby increasing the execution efficiency and success rate of encirclement missions.