The photos taken on rainy days suffer from degraded visual quality, posing challenges to security-critical tasks such as object detection in surveillance and autonomous systems. Existing image deraining methods often combine Transformer and CNN architectures for performance gains. However, due to the inherent discrepancy of feature extraction between CNN and Transformer, these methods struggle to model complex dependencies between rain streaks and background. To address this, we propose a dual-channel fusion-based network for image deraining, called DF TransNet. Specifically, we design a Dual-channel Fusion Transformer (DF Transformer) as the encoder, employing a dual-path structure to separately capture global semantics and local details, with cross-connections to enhance feature interaction. To deeply fuse features of different granularities, we adopt a fusion module that integrates features from channel, spatial, and frequency dimensions. Additionally, we introduce a mixed-scale gated feed-forward network to improve robustness to varying rain patterns. In the decoder of DF TransNet, a Region Transformer Cascade (RTC) is designed to distinguish the rain-affected and rain-unaffected regions by masking mechanism for targeted feature reconstruction. Extensive experiments on public datasets confirm that our method achieves state-of-the-art deraining performance, offering practical benefits for vision-based security applications in harsh weather.