In recent years, parameter-efficient fine-tuning methods have gained attention as an alternative to full fine-tuning of pre-trained language models for transfer learning. Such methods can alleviate the problem that full fine-tuning requires updating and storing all the model parameters for different tasks, showing comparable performance with much fewer tuned parameters. However, their effectiveness in the context of multilingual natural language processing remains underexplored. This paper evaluates the performance of two representative parameter-efficient fine-tuning methods, prefix-tuning and LoRA, on multilingual abstractive summarization, comparing them with traditional full fine-tuning approach. Our comprehensive analysis highlights the trade-offs between efficiency and performance, providing benchmarks to standardize evaluation in this domain. Additionally, we delve into an in-depth examination of prefix-tuning, particularly under few-shot conditions, uncovering insights into its efficacy and offering guidance for optimizing its performance in multilingual environments. Our findings contribute to a deeper understanding of the applicability and benefits of parameter-efficient fine-tuning methods in multilingual natural language processing. This research aims to inform future developments and encourage the adoption of parameter-efficient fine-tuning techniques in multilingual contexts, ultimately enhancing the performance and scalability of pre-trained language models. We have made our source code publicly available at https://github.com/sgallon-rin/peft-mas .