Transfer learning, by leveraging knowledge from pre-trained models, has significantly improved the performance of downstream tasks. However, as deep neural networks continue to scale, full fine-tuning poses substantial computational and storage challenges in resource-constrained environments, limiting its practical adoption. To address this, parameter-efficient fine-tuning (PEFT) methods have been proposed to reduce computational complexity and memory requirements by updating only a small subset of parameters. Among them, matrix decomposition-based approaches such as LoRA have shown promise, but often struggle to fully capture the high-dimensional structural characteristics of model weights. In contrast, high-order tensors offer a more natural representation of neural network parameters, enabling richer modeling of multi-dimensional interactions and higher-order features. In this paper, we propose tCURLoRA, a novel fine-tuning method based on tensor CUR decomposition. By stacking pre-trained weight matrices into a third-order tensor and applying tensor CUR decomposition, our method updates only the compressed tensor components during fine-tuning, thereby substantially reducing both computational and storage costs. Experimental results show that tCURLoRA consistently outperforms existing PEFT approaches on medical image segmentation tasks. The source code is publicly available at: https://github.com/WangangCheng/t-CURLora .