In recent years, compression artifact removal technique has garnered significant attention in the field of image compression. However, existing artifact removal methods always adopt black-box networks for image enhancement. Although some explicable networks have better capability on reducing compression artifacts, they attach no importance to the role of image decomposition on network architecture design. To this end, we build a decomposition-driven two stages artifact removal framework for compressed images, which is composed of L1-norm and Low-Rank Constrained Structure Enhancement (LRC-SE) sub-optimization model and L1-norm Constrained Structure-Texture Enhancement (LC-STE) sub-optimization model. These two models can be unfolded into two explicable networks, named LRC-SE network and LC-STE network. The LRC-SE network is proposed to enhance the structure map, which is used as the initialization in the LC-STE network. Additionally, we design a low-rank latent representation block to decompose high-dimension features for effective feature extraction and redundancy reduction. Extensive experimental results demonstrate that the proposed method achieves superior performances for compression artifact removal in terms of PSNR, PSNR-B, and SSIM.