High-resolution (HR) remote sensing is essential for remote sensing image interpretation, but challenges in super-resolution (SR) stem from scale and texture differences within images, neglecting high-dimensional detail extraction and long-range dependencies among multi-dimensional features. Addressing this, we propose TDSNet, a enduring memory self-learning multi-level Transformer Network for remote sensing image super-resolution (RISSR). The utilization of the similarity balancing module and memory gated group establishes connections between mixed-scale information, while also possessing enduring memory across various receptive fields. Shallow and deep-level data fuse in the transformer, employing a dual learning strategy, enhancing reconstruction through a constrained mapping process with a loss function. This transforms the ill-posed problem into a well-posed one. Attribution analysis with the LAM method reveals TDSNet’s efficacy in capturing content information. Experiments on NWPU-RESISC45 and AID datasets demonstrate TDSNet’s superior performance in remote sensing image super-resolution compared to other methods.