HeteroFormer: A Structure Aware Heterogeneous Graph Transformer
摘要
The emergence of the Graph Transformer (GT) as a novel architecture has demonstrated remarkable performance across various graph mining tasks. However, despite its successes, most existing GT models are tailored for learning node representations within homogeneous graphs. This limitation become particularly evident when attempting to learn representations within heterogeneous graphs, which encompass diverse node and edge types. Conventional graph Transformers typically treat nodes as independent tokens and aggregate them into a single lengthy sequence for training Transformer models. This aggregation strategy poses scalability challenges, especially for large graphs, due to the quadratic complexity in self-attention computation relative to the number of nodes. The area of Graph transformers on heteregeneous graph structured data is currently under-explored, most notable being HINormer. However, HINormer fails to capture global structural information. To address the above challenges, we come up with - HeteroFormer: 1) A metapath-based Hop2Token approach is introduced to aggregate hop-wise structural information, which also includes a global structural token for capturing global structural information. 2) Attention bias is introduced to induce a sense of distance in the calculation of attention weights. Extensive experiments on popular heterogeneous graph datasets such as IMDB, FreeBase, AMiner show that HeteroFormer outperforms the current state-of-the-art method HINormer by an average of 1.92% and traditional message-passing GNNs by an average of 8.23%, validating its efficacy in node classification tasks.