The increasing amount of unorganized text information being produced has led to the use of summarization in a variety of fields, including news, law, and medical domain. More research is being done on creating summaries that are more human-like and fluent, as well as on improving summarization systems and datasets for testing and fine-tuning natural language processing models. This study delves into this area, focusing on the abstractive text summarization capabilities of two prominent transformer models: T5-base and Llama 3 with Parameter-Efficient Fine-Tuning (PEFT). We evaluated the models using benchmark summarization datasets, CNN/Daily Mail (CNNDM) and Extreme Summarization (XSUM) datasets, to provide a comprehensive assessment of their performance. Using ROUGE scores, we analyze model competency, and from the experimental results, it is vivid that Llama version 3 (8 billion parameters) with the PEFT technique, specifically Low-Rank Adaptation of Large Language Models (LoRA), outperforms in abstractive text summarization. The application of LoRA freezes the pre-trained model weights and integrates trainable rank decomposition matrices into each transformer layer present in the architecture, significantly reducing the number of trainable parameters which is further resultant into the reducing the GPU memory requirement.