Transformer-based pre-trained models have achieved breakthrough results in deep learning, and fine-tuned models can achieve good performance on a wide range of tasks. However, it is challenging to directly deploy these models and fine-tune them with local data, on resource-constrained devices. To address this limitation, this paper introduces a framework for the distributed deployment of transformer-based models on the device-edge and outlines the process for efficient fine-tuning. By fine-tuning partial parameters of the model instead of adjusting all parameters, the computation and memory overhead in the fine-tuning task is optimized. The communication pressure is reduced by freezing and masking a portion of the neurons in the layer. Evaluation results indicate that this simple and efficient fine-tuning method has little impact on the model accuracy.