Parameter-Efficient Fine-Tuning (PEFT) is an attractive method for pre-trained large language models, among which Low-Rank Adaptation (LoRA) is an efficient one. To address the low-rank bottleneck of LoRA, MELoRA has recently been proposed. We found that leveraging information fusion between vector subspaces can further enhance the generalization capability of MELoRA. Thus, we propose Multi-head Low-Rank Adaptation (Mh-LoRA), Mh-LoRA decomposes LoRA’s dimensionality reduction and expansion matrices into multiple low-rank sub-matrices, each corresponding to a “head”. We then introduce a square matrix in the low-dimensional space to fulfill the role of the Output matrix in Multi-head Attention, effectively integrating information from different vector subspaces while avoiding the introduction of a large number of trainable parameters. Cross-modal image-text retrieval, instruction following, and mathematical reasoning tasks are tested. Experimental results show that compared to MELoRA, Mh-LoRA achieves a 2% performance improvement with only half parameters on image-text retrieval; 3% performance improvement with comparable parameter size on instruction following; 4% with the same number of parameters on mathematical reasoning. Furthermore, Mh-LoRA is more effective in mitigating catastrophic forgetting than MELoRA and LoRA.