RGBT tracking aims to achieve robust tracking by comprehensively utilizing the features of visible and thermal infrared modalities. Existing methods achieve multimodal interaction and fusion by designing complex fusion modules. However, due to their adoption of intermediate fusion or late fusion strategies, these methods result in inefficient tracking, which may limit their application in real-time tracking scenarios. In this paper, we propose an efficient single-stream RGBT tracking framework based on Early Fusion and Adaptive hierarchical knowledge Distillation, termed EF-AD. We design an early fusion strategy to improve tracking efficiency and reduce model complexity. Additionally, an adaptive hierarchical knowledge distillation strategy is devised to ensure the tracking performance of the student model. In particular, we design a early fusion module for performing early fusion of multimodal features after the encoding layer. In addition, we record the intermediate layer features and response map features of the teacher network, and compute the tracking losses for the two modal branches separately as indicators for adaptive fusion. Subsequently, the teacher features are adaptively fused to guide the feature learning of the student network in various scenarios. Extensive experiments on two popular RGBT tracking datasets demonstrate that our method significantly reduces computational complexity and the number of parameters while only incurring a slight decrease in accuracy, achieving an inference speed of 72.6 FPS.