Federated Learning is an emerging machine learning technology proposed by Google in 2016, which allows for collaborative training across multiple devices without the need to collect participants’ data. This approach helps protect user privacy. Using pre-trained models as the initialization for federated learning algorithms has become common in recent work. However, many efficient methods for fine-tuning pre-trained models under centralized learning setting seem to have not been applied in these works. We notice that the two-stage fine-tuning (FT) method, linear probing then fine-tuning (LP-FT), performs well in centralized transfer learning, so this paper expands it to federated learning problems. This paper proposes a new federated learning method called FedLP + FT. The method adopts a two-stage strategy: in the first stage, the linear head of the model is trained using linear probing; in the second stage, fine-tuning update the entire model following the traditional federated learning approach. The experimental results show that the FedLP + FT has potential to reduce communication and compute costs while achieving better model performance.