Text-Attributed Graphs (TAGs) seamlessly integrate textual data with graph structures, presenting unique challenges and opportunities for jointly modeling text and graph information. Recent advancements in Large Language Models (LLMs) have significantly enhanced the generative and predictive capabilities of text modeling. However, existing graph models often fall short in capturing intricate node relationships, as their edge representations are typically limited to scalar values. In this paper, we introduce SheaFormer, a novel method that encodes rich and complex relational information between nodes as edge vectors. During the message-passing phase, SheaFormer aggregates both neighbor node representations and edge vectors to update the central node’s representation, eliminating the need to fine-tune the LLMs on the text-attributed graph. Specifically, for a given TAG, SheaFormer is trained to minimize the prediction errors of the LLM in forecasting the next word in node text sequences. Furthermore, we enhance SheaFormer’s performance by incorporating prompt-based fine-tuning techniques. Once trained, SheaFormer can be seamlessly adapted to various downstream tasks. Extensive node classification experiments across multiple domains demonstrate that SheaFormer consistently achieves state-of-the-art performance, validating its effectiveness in capturing complex relationships within TAGs. Additionally, we conduct ablation studies and scalability analyses to ensure the robustness and applicability of our approach.