The significant delay in the inference process associated with large vision transformers (ViTs) limits their use in real-time applications; however, model compression can mitigate this issue. Post-Training Quantization (PTQ) is a time-efficient method that requires minimal calibration data, eliminating the need for end-to-end retraining. Current PTQ schemes suffer from accuracy degradation at low-bit precision and primarily optimize the quantization scale globally using gradient-based methods. To address this, we propose a novel PTQ method that incorporates reconstruction optimization and scale perturbation update specifically for ViTs. We first calculate the output loss of the corresponding blocks one by one between the full-precision model and the initial quantized model, updating the gradient and reconstructing the model accordingly. Subsequently, we introduce a perturbation to the quantization scale of each attention block, with the updated scale determined through a genetic algorithm. Comprehensive experiments on large-scale classification datasets demonstrate that our method outperforms existing PTQ techniques, particularly at low-bit settings.