Vision-based real-time pose tracking of endoscope is significant for navigation and automation of endoscopy. In this work, a deep learning-based framework is proposed for vision-based real-time pose tracking of nasal endoscope based on endoscopic images. Specifically, an attention-based module is introduced into the feature encoder to extract and integrate multi-dimensional and multi-scale information. Furthermore, a novel pose encoder is proposed to extract more complicated representation from the feature map for pose vector prediction. The proposed method is compared with state-of-the-art methods on the dataset, which consists of endoscopic videos and endoscope poses obtaining by an optical tracking system. The experimental results demonstrate that the proposed network achieves better performance than state-of-the-art works for real-time nasal endoscope pose estimation. The average tracking error of our method is 5.78 mm, while the relative translation error and rotation error are respectively 0.48 mm and 0.22 degree on average. In addition, the average inference speed of our network is 46 fps, which achieves the real-time requirement.