Medical image segmentation task plays an important role in areas such as clinical care and medical atlas construction, but existing 2D segmentation models often face challenges in 3D organ or tissue segmentation tasks, and 3D models account for a large computing resource proportion. Therefore, we consider 3D tissue organs as video sequences and use the given first layer of tissue organ boundaries as a sequence of query points to predict matching points on subsequent slices. Based on this idea, we propose a point tracking architecture optimised based on the joint point tracking model CoTracker, which compensates for the shortcomings of CNN-based architectures in global feature information extraction and improves the robustness of the overall image features by incorporating the robust, strongly generalisable DINOv2 encoder. The model uses semantic features extracted by the self-supervised learning foundational model DINOv2-ViT for feature fusion with ResNet, and optimises them with a fine-tuning strategy based on the CoTracker weights. Specifically, we introduce the channel attention mechanism to make full use of Vision Transformer’s feature recognition capability, and achieve feature optimisation by filtering high-weighted channels, thus improving the accuracy of several evaluation metrics in the point tracking domain. Extensive evaluation results show that our approach not only greatly saves training resources, but also efficiently improves tracking accuracy and has good generalisation in the field of medical image segmentation.