In clinical practice, accurate polyp segmentation provides important information for the early detection of colorectal cancer. Benefiting from the advancement of deep learning techniques, various neural networks have been developed for polyp segmentation. However, most state-of-the-art methods have suffered from the challenge of precisely segmenting polyps with clear boundaries. To tackle this challenge, in this paper, we propose a novel and effective cascade boundary-guided attention network based on an encoder–decoder framework. Specifically, instead of just using the addition of shallow and deep features, the fine-grained boundary information is explicitly introduced into the skip connection of encoder and decoder layers to achieve accurate polyp segmentation. Moreover, the cascade refinement strategy is utilized into the multi-stage enhancement of boundary features to progressively produce better predictions. Extensive evaluations on five public benchmark datasets show that our method outperforms state-of-the-arts on various polyp segmentation tasks. Further experiments conducted on the cross-dataset (training on one dataset and testing on another dataset) validate the generalization ability of the proposed method.