Image deraining aims to transform a rainy input image into an image of high quality. Transformer-based techniques have demonstrated remarkable efficacy in image deraining because of their capacity to represent non-local information, a crucial element for high-quality image reconstruction. Our findings indicate that most transformers used for feature aggregation nowadays leverage all token similarities between query-key pairs. If the query tokens differ from the key tokens, the quality of the recovered image is compromised by the self-attention values derived from these tokens if these are considered during feature aggregation. For this purpose, we propose the Progressive Sparse Transformer Network (PSTNet), which progressively learns to restore degraded inputs by retaining the helpful self-attention information during feature aggregation and discarding the remaining values which obstruct the restoration. Transformer blocks help to capture interactions between distant pixels. Thorough experimental outcomes on widely used benchmarks show that the suggested approach performs better than most existing techniques.