Image classification is the core and fundamental problem in the field of computer vision. Many high-level vision tasks are highly dependent on ability of image classification. Transformer structure has been proved better performance than convolutional neural network when applied to the field of image classification. Although Transformer structure has been extensively studied, there still lacks investigation on the privacy protection when applying it to visual applications. In this paper, we propose a privacy-preserving Transformer based image classification framework. We employ a modified additive secret sharing scheme to support the calculation of negative fixed points. Then, we use mathematical methods to design secure division protocol and exponential protocol, whose computation cost and communication overhead is significantly reduced compared with previous secure protocols. Finally, we securely construct the essential network modules in Transformer structure framework through the proposed sub-protocols, so as to ensure the correct calculation of the network. The experiment results on public datasets show that the proposed secure framework achieves practical efficiency and accuracy.