The area of graph clustering has received significant attention due to its diverse applications in various domains such as recommender systems and bioinformatics. However, most of the existing methods (i) lack the effective use of high-order neighbor information due to the message passing mechanism of graph neural networks (GNNs), and (ii) ignore high-order modularity information. To address these shortcomings, we propose a new clustering method named High-Order Structure Enhanced Graph Clustering Network (HSEGC). Specifically, HSEGC integrates a multi-head attention fusion graph convolution module to formulate a weighted adjacency matrix, thereby enriching high-order neighbor information. Moreover, HSEGC incorporates a modularity maximization module to capture high-order modularity information effectively. Comprehensive experiments conducted on four commonly used benchmark datasets demonstrate the efficacy of HSEGC in leveraging high-order information for deep clustering.