Unsupervised Video Object Segmentation (UVOS) aims to segment primary objects in videos without relying on manual annotations, yet it remains highly challenging due to the lack of supervision and the need for robust adaptation across diverse scenes. In this work, we present a two-stage framework that integrates training-time training and test-time training to address these challenges efficiently. During training, a lightweight Lie Group adapter is inserted into a pretrained encoder, forming the Foundational Representation Module (FRM), which learns task-relevant knowledge while preserving general representations and minimizing parameter updates. At test time, we introduce a Specialized Refinement Module (SRM), which adapts to unseen domains by updating its own adapter and decoder using self-supervised signals, while keeping the FRM frozen to avoid catastrophic forgetting. The outputs from both modules are fused at inference to produce accurate and robust segmentation masks. Extensive experiments on standard UVOS benchmarks demonstrate that our method achieves state-of-the-art performance with minimal computational overhead, making it especially suitable for real-world and dynamically evolving video environments.