<p>Trajectory prediction for autonomous driving requires a careful balance between prediction accuracy, computational efficiency, and scalability. However, existing approaches often struggle to achieve high accuracy without incurring substantial computational overhead. To address these limitations, we propose Graph Contextual Fusion Refinement Network (Graph-CFRNet), a graph-based framework built upon instance-centric scene representations. Graph-CFRNet constructs sparse yet semantically meaningful interaction graphs and employs an Edge Fusion Graph Attention Layer (EFGAL) to enhance the feature representations of both dynamic agents and lane elements, effectively modeling intra-modality and cross-modality interactions. To incorporate global scene context, we introduce the Contextual Fusion Refinement Decoder (CFRD), which aggregates instance features within a unified coordinate system and applies attention to extract agent-specific contextual cues. This enables each agent to refine its trajectory predictions using comprehensive scene information. Experiments on the Argoverse 1 and Argoverse 2 benchmarks demonstrate that Graph-CFRNet achieves competitive prediction accuracy while significantly reducing computational cost and training complexity, making it well-suited for real-time deployment in dense traffic scenarios.</p>