Polyp segmentation has greatly benefited doctors in early diagnosing and evaluating colorectal cancer since it took advantage of modern deep-learning methods. The precision and robustness of polyp segmentation models have gradually improved in recent years despite hard challenges such as tiny polyps or edge missing. However, these problems are not alleviated completely and also exist in other complex tasks of medical image processing. Our study proposes a new method called contour-enhanced segmentation to enhance the capability of capturing the exact edge of polyps for accurate segmentation, which is important not only in this task but also in other segmentation tasks. While we are not attempting to build an end-to-end model to resolve the problem, we first train a model to detect the contour of the polyps and then use transfer learning for the downstream segmentation. We also devise a simple but efficient method to infer the contour of the polyps. By leveraging cutting-edge advancement in the computer vision field including attention mechanism and MetaFormer, our model achieves competitive results to other state-of-the art methods and even outperforms them in some cases on three popular datasets CVC-ClinicDB, Kvasir-SEG and ETIS-PolypDB.