Open-World Weakly-Supervised Object Localization (OWSOL) targets the recognition and localization of both known and novel categories in open-world situations. The pioneering work primarily tackles the problem by proposing a generalized representation learning paradigm, which lacks targeted optimization for Vision Transformers (ViT). We argue that long-range visual dependencies of ViT are specialized in complete perception of objects. To better adapt ViT into OWSOL, in this work, we propose a Token-level Contrastive Learning (ToCL) framework. It mainly contains supervised contrastive learning on labeled data and, semantics-driven token-level contrastive learning on labeled and unlabeled ones. Specifically, contrastive learning is performed on both class and patch tokens, which learns complementarily for fine-grained semantics. Besides, tokens of foreground and background are learned to distribute apart by contrast. The above operations potentially enable self-attentions of ViT to accurately and completely focus on the target object regions. Extensive experiments on ImageNet-1K, iNatLoc500, and OpenImages150 datasets show that our method outperforms the state-of-the-art methods by a large margin. Code will be released.