Graph invariant learning aims to acquire invariant node representations across various environments, achieving substantial success in addressing Out-of-Distribution (OOD) generalization for graph learning tasks. As obtaining environment splits on graphs is typically costly, most graph invariant learning methods heavily depend on inferring the underlying environments to learn invariant node representations. Due to the high heterogeneity of graph data without explicit source labels, existing environment inference methods cannot simultaneously satisfy the requirements of diversity and similarity. To address this challenge, we propose an approach called sOft environment inFerence with Test-timE adaptatioN, abbreviated as OFTEN, which enables us to perform graph invariant learning without any predefined environment split or partition information. The intuition is to enhance the diversity among environments while preserving the original graph topology. Extensive experiments on several graph OOD benchmarks demonstrate the consistent superiority of OFTEN across all settings.