<p>Long-document summarization remains a challenging task due to complex discourse structure, semantic sparsity, and long-range dependencies. Existing transformer and graph-based models often struggle to capture multi-level semantic interactions effectively, while abstractive large language model (LLM) pipelines suffer from elevated hallucination rates on documents with thousands of tokens. To address these limitations, we propose <b>CAGNN-Sum</b>, a novel context-aware heterogeneous graph neural network-based extractive summarization framework. Our approach constructs a heterogeneous graph where sentences, named entities, and BERTopic-derived latent topics are treated as distinct node types, each capturing different semantic granularity. These nodes are connected via typed relational edges encoding semantic associations such as co-reference, topical alignment, and discourse similarity. A customized heterogeneous graph neural network (HGNN) performs type-aware message passing, incorporating a relation-aware attention aggregation mechanism that adaptively weights inter-type and intra-type interactions. We introduce a multi-level graph attention pooling (ML-GAP) strategy with input-conditioned scalar gates that hierarchically scores sentences by leveraging attention distributions from entity and topic nodes. Experiments on arXiv and BillSum datasets show that CAGNN-Sum achieves ROUGE-1 scores of 47.62% and 46.48%, significantly outperforming strong baselines including G-Seek-2 and HeterMDS. Additionally, BERTScore evaluation confirms that the semantic quality of generated summaries surpasses that of token-overlap metrics alone. Retrained ablations with Wilcoxon signed-rank testing confirm that ML-GAP pooling (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">$p{<}0.001$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo><</mo> <mn>0.001</mn> </mrow> </math></EquationSource> </InlineEquation>) and encoder depth are the architecturally critical components, while cross-domain transfer (arXiv<InlineEquation ID="IEq2"> <EquationSource Format="TEX">$\rightarrow $</EquationSource> <EquationSource Format="MATHML"><math> <mo stretchy="false">→</mo> </math></EquationSource> </InlineEquation>BillSum, ROUGE-1 = 42.07 without retraining) demonstrates structural generalization.</p>

Context-aware heterogeneous graph neural networks with attention-based pooling for long-document summarization

期刊论文
发表日期：

2026年6月16日

查看全文

Syed Wajahat Ali Bukhari,
Yubo Yan

摘要

Long-document summarization remains a challenging task due to complex discourse structure, semantic sparsity, and long-range dependencies. Existing transformer and graph-based models often struggle to capture multi-level semantic interactions effectively, while abstractive large language model (LLM) pipelines suffer from elevated hallucination rates on documents with thousands of tokens. To address these limitations, we propose CAGNN-Sum, a novel context-aware heterogeneous graph neural network-based extractive summarization framework. Our approach constructs a heterogeneous graph where sentences, named entities, and BERTopic-derived latent topics are treated as distinct node types, each capturing different semantic granularity. These nodes are connected via typed relational edges encoding semantic associations such as co-reference, topical alignment, and discourse similarity. A customized heterogeneous graph neural network (HGNN) performs type-aware message passing, incorporating a relation-aware attention aggregation mechanism that adaptively weights inter-type and intra-type interactions. We introduce a multi-level graph attention pooling (ML-GAP) strategy with input-conditioned scalar gates that hierarchically scores sentences by leveraging attention distributions from entity and topic nodes. Experiments on arXiv and BillSum datasets show that CAGNN-Sum achieves ROUGE-1 scores of 47.62% and 46.48%, significantly outperforming strong baselines including G-Seek-2 and HeterMDS. Additionally, BERTScore evaluation confirms that the semantic quality of generated summaries surpasses that of token-overlap metrics alone. Retrained ablations with Wilcoxon signed-rank testing confirm that ML-GAP pooling ( $p{<}0.001$ $p < 0.001$ ) and encoder depth are the architecturally critical components, while cross-domain transfer (arXiv $\rightarrow $ $\to$ BillSum, ROUGE-1 = 42.07 without retraining) demonstrates structural generalization.