Parameter-efficient fine-tuning (PEFT) has become a standard approach for transfer learning of pre-trained models by freezing most of the parameters and selectively updating a small subset for fine-tuning. However, existing parameter-efficient fine-tuning methods for VLMs typically focus on learning task-specific parameters for particular tasks, which rarely take advantage of the rich shareable knowledge across different visual-language tasks. The unified multimodal input-output architecture makes pre-training on large-scale corpora possible, thereby also facilitating joint downstream multitask fine-tuning accordingly. In this paper, the proposed method, named VL-MPFT, mainly contributes to multitask parameter-efficient fine-tuning for visual-language pre-trained models. Specifically, VL-MPFT introduces a task-adaptive masking matrix with low-rank decomposition for the diverse requirements of tasks while sharing most parameters for multitask learning and a dimensionality-reducing cross-attention module for modality fusion. Our experiments on VQA, image caption, and visual reasoning tasks validate the state-of-the-art average PEFT performance provided by VL-MPFT. We also conduct ablation experiments to assess the contributions of individual modules in VL-MPFT and validate the effectiveness and superiority of the proposed modules.