Handling lengthy documents for multi-label document classification for downstream clinical tasks is quite challenging. With the advent of pre-trained transformer models like BERT and its variants, several studies have put forth approaches that leverage pre-trained models tuned for specific end target datasets. While such models perform well for short text classification problems, handling long documents, especially in the healthcare domain, has its challenges. The LongFormer model is one such effort, which is trained on long input sequences while incorporating a sparse attention mechanism following a windowing operation similar to convolutional networks. In this work, we propose a Parameter Efficient Tuned LongFormer model (PET-Former), with a focus on medical code prediction from long clinical documents. It enables the relearning of only data-specific layers in the over-parameterized transformer network by retaining the base or original pre-trained features for better convergence. This also helps overcome the issues of training the over-parameterized large transformer model by optimizing its data-specific parameters. The proposed approach was evaluated on the standard MIMIC IV dataset for classifying the top fifty ICD-10 codes, demonstrating benchmarking performance while maintaining low computational overhead.