ViT deployment on edge devices is hindered by their significant computational demands and limited platform acceleration support. Network quantization emerges as a promising solution, with two main paradigms: post-training quantization (PTQ) and quantization-aware training (QAT). While PTQ is efficient and simple, it struggles with ultra-low-bit quantization. Conversely, QAT offers potential in lower-bit scenarios but at a high demand for training data, exacerbated by the complex architecture of ViTs. To overcome the trade-off between quantization efficiency and high performance for low-bit quantization, we propose the Vision Transformer Quantization-Aware Efficient Training (ViT-QAeT) approach. ViT-QAeT incorporates coreset selection to optimize the training dataset, thereby bridging the gap between PTQ’s efficiency and QAT’s precision. However, a significant challenge of catastrophic forgetting arises during the coreset update process for training quantized ViTs. To combat this, we propose a novel Fisher Matrix Regularization (FMR-QViT) methodology. FMR-QViT employs a Fisher Matrix Regularization module, developed through Bayesian analysis, to preserve and leverage knowledge from previous coresets, ensuring an effective and practical training process. Extensive experiments validate the efficacy of FMR-QViT in reducing training time and computational demands while maintaining high performance in ultra-low-bit quantization scenarios.