Recently, integrating eye-tracking techniques and texts into image-based disease classification has gained traction. To address the unmet needs such as heterogeneous data alignment, information propagation and aggregation, and expert knowledge embedding, we propose an innovative expert-guided Graph Neural Network (GNN) that uses radiologists’ eye-gaze data and transcribed audio reports with X-ray images during training. By distilling expert knowledge from gaze data and diagnosis reports, our GNN can achieve high accuracy using only X-ray images during inference. This approach provides a robust framework for disease diagnosis, embedded with the radiologists’ insights, addressing challenges in aligning heterogeneous data, propagating local information for global decisions, and leveraging expert knowledge effectively. Additionally, the attention maps on X-ray images which are generated from the GNN model visualize the Region of Interest (ROI) for the diagnosed disease. Evaluated on two benchmark chest X-ray datasets, the proposed method outperforms state-of-the-art X-ray image classification methods.