Image de-fencing refers to the removal of fence structures from an image thereby creating a visually attractive synthetic image rendered with appropriate colors in place of the fence regions in the input image. In this paper, we focus on developing a lightweight deep neural model for image de-fencing that can be conveniently used on devices with less memory and processing power such as smartphones and other portable devices. Specifically, we present a response-based knowledge-distilled student model termed DefenceLite which is trained by transferring knowledge from an effective GAN-based teacher de-fencing model. DefenceLite has fewer parameters (approximately \(80\%\) less) than the teacher model and, thus, has a lower response time. It is trained with an extensive dataset consisting of pairs of fenced and corresponding de-fenced images using a combination of adversarial loss and L1 loss. According to experimental findings, DefenceLite is capable of providing visually pleasing de-fenced outputs that are comparable with the teacher model outputs but at the cost of significantly less processing time. Qualitative and quantitative comparative studies with recent deep learning-based de-fencing techniques demonstrate that despite being much lighter, DefenceLite has the potential to achieve results comparable to the recently developed deeper de-fencing models.