Automatic accurate identification of a lymph node (LN) in digitised Haematoxylin/Eosin stained tissue sections from surgical resection specimens is challenging due to the structural variability of LNs, the presence of metastatic and non-metastatic regions, therapy-induced tissue changes, and the variability in tissues surrounding the individual LN, including fat and part of the organ wall with or without the primary tumour. These complexities pose challenges for accurately determining LN size and microarchitecture in an automated manner, underscoring the need for robust methods to unlock their prognostic potential. To address these challenges, we introduce GTFuse, a novel deep learning framework that combines SegFormer-based multi-class tissue segmentation with a graph convolutional network (GCN) for a more accurate LN segmentation. GTFuse is able to distinguish between primary tumour regions and tumour regions within LNs by multi-class segmentation, where tumour is a single class, based on local appearance with contextual information provided by a graph neural network. We evaluated GTFuse on independent internal and external datasets, demonstrating a performance comparable to or surpassing existing state-of-the-art methods, including foundation model MedSAM. Notably, GTFuse significantly reduced false positive detection in slides without LN tissue, thereby improving specificity and minimizing classification errors. By addressing tissue heterogeneity and enhancing segmentation accuracy, GTFuse presents a promising solution for both clinical and research applications in digital pathology.