Deep neural networks have demonstrated superior performance in various image segmentation tasks. However, their decision-making process lacks a clear mathematical explanation, and complex structures and massive parameters often result in extensive training data and computational resources. Conversely, traditional variational model-based methods offer a robust theoretical foundation but can be challenging due to initialization sensitivity and manual parameter tuning requirements. In this paper, we present an end-to-end variational image segmentation network framework based on the unfolding of the Potts model named PottsNN. It is designed under the guidance of the iterative process of traditional Potts models, which provide the mathematical interpretation of the network. The proximity operator is introduced using the HQS (Half Quadratic Splitting) method to solve the Potts model. Based on corresponding mathematical solutions, we design a UNet-like architecture for learning regional features. During the upsampling phase, the regularization term module incorporates the Fields of Experts (FoE), and the gradient operator is learned through a trainable discrete cosine transform (DCT)-Gaussian convolution. The activation is achieved using a soft threshold formula (STF). Experiments demonstrate that our model can efficiently learn features from small datasets and achieve significant accuracy improvements. Compared to classical image segmentation networks, PottsNN reduces parameters by 72% to 98%. Furthermore, our model is robust to noise interference. This work combines the benefits of deep neural networks and traditional variational models, providing potential directions for network design and future improvements based on model unfolding.