<p>Foundation models have garnered increasing attention due to their superior discriminative ability. Visual prompt tuning (VPT) is a popular technique for fine-tuning foundation models on downstream tasks, which only learns a small set of parameters during adaptation while maintaining the pre-trained parameters as frozen, retaining high accuracy. However, we observe that when the data of downstream tasks follow a long-tailed distribution, the fine-tuned model presents poor discriminative ability on tail classes. In this paper, we analyze the limitations of VPT on imbalanced datasets and find that the learned prompts are heavily influenced by the head classes, which leads to tail classes being underrepresented. Although prompt tuning can provide primary discriminative ability, it meanwhile hurts the uniformity of tail classes and thus prevents further improvement. Furthermore, we find that the pre-trained foundation models contain sufficient information, which can help the learning of tail classes. Thus, we propose the Uniformity Preserving Transfer (UPT) method to improve the representation of tail classes by transferring the uniformity from the foundation model to the fine-tuned model. UPT can make features more uniform in the feature space and therefore enhance the discriminative ability. Experimental results demonstrate the effectiveness of our method.</p>