Vision Transformers (ViTs) have demonstrated remarkable success in computer vision. However, the lack of inductive biases makes it particularly challenging to train ViTs on small datasets. Two methods are shown to mitigate this: 1) Self-Supervised Learning and Fine-Tuning sequentially (SSL + FT); 2) joint optimization of the Self-Supervised Auxiliary Task (SSAT) and downstream task. In this work, we systematically investigate the effectiveness of SSAT in both training from scratch and fine-tuning settings, further revealing that its success is closely tied to the design of the self-supervised objective within the joint optimization. Specifically, grounded in Information Bottleneck (IB) theory, we show that appropriately constructed self-supervised tasks can provide dual inductive biases that encourage local attention regularization (via compression) and rich semantic encoding (via reconstruction). Motivated by this, we propose a novel multi-view Feature-Guided Consistent Reconstruction (FGCR) method that accelerates learning minimal sufficient representations in IB, allowing ViTs to converge from scratch in small datasets quickly. Our evaluations across diverse datasets demonstrate that FGCR outperforms previous SSAT methods and yields additional gains over SSL + FT methods. Code is available at https://github.com/JAMESKirkQI/FGCR .