Since the segmentation of regions of interest from medical images is significant for doctors, current researches are in pursuit of high segmentation precision. However, the utilization and fusion of multi-scale hierarchical feature maps is unsatisfactory. In this paper, we propose a fine-grained recurrent network (FRNet) for medical image segmentation via vector field guided refinement, which is based on the encoder-decoder structure. In the proposed network, the encoder utilizes CNN to extract multi-scale feature maps. Then, in the decoder, for making better use of the multi-scale semantic features and obtaining finer features, we design a new fine-grained recurrent unit for refining feature maps and score maps. Additionally, vector field embedded in the decoder guides the improvement of upsampling accuracy and rectification of edge segmentation. Experimental results on four datasets demonstrate that the proposed FRNet not only improves the segmentation precision, but also flexibly works on different CNN-based backbones.