Transformer-based trackers have recently demonstrated remarkable performance in the visual tracking community. However, leveraging rich information across temporal frames is difficult for conventional visual Transformers due to the quadratically increasing complexity of the self-attention computation, thus limiting the performance potential of Transformer-based trackers. In this paper, we propose a more efficient one-stream tracking framework by integrating temporal context-aware token learning and scale-adaptive token pruning. Specifically, we employ a temporal context-aware encoder that simultaneously enhances feature learning and relation modeling by mutually interacting the inherent template, dynamic templates, and the search region solely through self-attention. To balance the tracking accuracy and speed, a scale-adaptive token pruning module is proposed based on learnable scale attention, thus making the search token pruning more adaptable to different target scales. Furthermore, to handle multiple templates during inference and improve tracking robustness, we design a temporal template updating strategy that dynamically selects the templates capturing appearance variations of target objects. The results on five benchmarks show that our tracker obtains superior tracking performance while maintaining fast inference speed.