Although visual transformers have been initially applied in pose estimation, the existing pose estimation methods using visual transformers either employ heat map based representations or retain both pixel bins and keypoint tokens simultaneously. This results in redundant computational costs, leaving further room for optimization in the lightweighting of the model. In this study, by deeply decoupling the limitations of the paradigms of the currently popular pose estimation frameworks and the engineering adaptation bottlenecks of the visual Transformer, an architecture-innovative ViTRotPose model is proposed. This model demonstrates excellent capabilities in the pose estimation task in terms of flexibility, scalability, simplicity, etc. Specifically, the main structure is based on the collaborative optimization of the global attention mechanism of depth separable convolution and visual transformers, constructing a multiscale pyramid hybrid architecture. By reconstructing the standard self-attention layer in the traditional Transformer into a dynamic depth separable convolution module, the number of parameters of the model is significantly reduced while maintaining high accuracy. Through the rotational position coding module, it effectively solves the problem of false detection of key points in traditional methods in scenarios with dense crowds and severe occlusion. Then, pinwheel-shaped convolution is introduced, and an independent filling strategy is adopted to construct horizontal and vertical cross-perception paths, expanding the receptive field and enhancing the ability to extract human features. Experimental data show that ViTRotPose has achieved better improvement and balance in terms of accuracy and lightweight, providing a brand new paradigm for lightweight pose estimation models based on visual transformers.