Modular skill acquisition has emerged as a promising paradigm in multi-task parameter-efficient fine-tuning (PEFT), enhancing knowledge organization and task transfer. Building on this concept, we propose MMtuning, an advanced PEFT framework for multimodal large language models (MLLMs), enabling direct fine-tuning from pre-trained unimodal models. In MMtuning, we formulate a multimodal skill allocation matrix that concurrently learns with the multi-adapter skill inventory, enabling optimal skill allocation for input samples during task fine-tuning. Experiments on ScienceQA and Visual7W demonstrate that MMtuning achieves superior sample efficiency compared to existing PEFT methods with the equivalent parameter volumes.