In this paper, we address the challenge of single image reflection removal (SIRR), a crucial task in computer vision that involves eliminating undesirable reflections from images captured through glass surfaces. Current state-of-the-art methods typically rely on convolutional neural networks (CNNs) and often make certain assumptions about the appearance of reflections, which may not hold true in real-world scenarios. To overcome these limitations, we propose a novel Transformer-based approach, DereflectFormer, inspired by the Swin Transformer. Our architecture introduces a new module, the Depthwise Multi-Activation Feed-Forward Network (DMFN), which leverages depthwise convolution and a dual-stream ReLU-GELU activation function to enhance detail extraction capability. We also employ a synthetic dataset and a synthesis method for training, which allows our model to fully exploit the capabilities of Transformer architectures. Based on experimental results, we demonstrate that our approach performs better than state-of-the-art methods, providing more accurate and robust results in various real-world scenarios. Furthermore, our ablation studies reveal that each component of our architecture contributes significantly to its performance, offering valuable insights for future research in the field of single image reflection removal. The code and dataset is available at https://github.com/Agent76ow/DereflectFormer .