Although modern language models (LMs) demonstrate excellent performance in diverse text processing tasks, the substantial GPU memory required to load and infer these models can be prohibitive to users. To compress and accelerate LMs, various techniques, such as quantization, distillation, pruning, and low-rank factorization, are used. In this work, we focus on improving a method from the latter category, namely, a recent technique Fisher-Weighted Singular Value Decomposition (FWSVD). Despite its efficiency, FWSVD requires fine-tuning of the whole model on a downstream task. We introduce a simple, yet powerful, modification of FWSVD that enables compression of models previously unavailable with the original approach. By combining LoRA with FWSVD we demonstrate that low-rank-based compression can be achieved without storing the full gradients, sometimes even outperforming the original full fine-tuning. We evaluate our proposed approach on various NLP tasks, including NLU, NER, text summarization, and QA, showing its effectiveness compared to strong baselines.