The existing Document Image Rectification (DIR) methods are limited by poor generalization to complex degradation distributions. Moreover, they often suffer from large computational complexities. To address these limitations, we propose a simple but effective Dynamic Adaptive Aggregation Network (DAA-Net) to mitigate the noisy interactions of irrelevant areas and remove feature redundancy in both spatial and channel domains. To be specific, the proposed DAA-Net mainly consists of three key parts, including Adaptive Receptive Field Convolution (ARFConv), Hierarchical Sparse Adaptive Attention (HSAA) and Saliency Feature Refinement Mechanism (SFRM). Among them, ARFConv can adaptively eliminate parameter-sharing constraints while maintaining computational efficiency through assigning position-specific attention weights on input distorted images. HSAA adopts a two-branch self-attention paradigm, named sparse and dense. The sparse branch can filter out noisy interactions from irrelevant tokens. While dense branch can ensure sufficient information flow through the network for learning discriminative representation. Meanwhile, SFRM employs an enhance-and-ease scheme to eliminate feature redundancy in channels for further image quality improvements. With the cooperation of the above components, our proposed DAA-Net can effectively handle complex deformation degradations and achieve state-of-the-art results on several popular benchmarks.