Inspired by robustness and efficiency of human language processing, we present Mask-guided Visual Text Transformer (Mg-VTT), which incorporates the use of visual text representations and a mask mechanism into Vision Transformer, creating continuous vocabularies by processing visually rendered text using sliding windows, obtaining task-relevant embedding with mask operation and finally producing robust, word-level, and open-vocabulary representations for classification task. We evaluate the proposed Mg-VTT on Chinese radiology reports dataset Xiangya radiology reports and English radiology reports datasets MIMIC-CXR with gradient-weighted class activation mapping (Grad-CAM) as the mask. Experimental results prove that Mg-VTT model significantly improves performance and efficiency when compared with vanilla Transformer and Vision Transformer. Furthermore, radiologists tend to use their native language to write, which hinders the comprehensive use of radiology reports in different languages. Although the previous studies respectively provide effective solutions to the problems of Chinese and English radiology reports, they are all modeled for a single language, lacking of research on text representation unified modeling of Chinese and English reports. Finally, Mg-VTT provides a novel view and a concrete method towards generalizing data-intensive and large-scale vision models ( https://github.com/jzw1234/Mg-VTT ).