Edge detection aims to find visually distinctive edges or boundaries in input images. Edge detection has made significant progress with the help of deep Convolutional Networks (ConvNet). Most ConvNet-based edge detectors predict each pixel independently and ignore the inherent correlations between pixels. However, structured cues in input images are critical to learn a good edge detector. To this end, we propose a novel Iterative Residual Holistically-nested Edge Detection (IRHED) network. IRHED incorporates multi-scale features from the hierarchy of the network, and learns to iteratively refine the output boundary map in a deeply supervised manner. In this way, global structural cues, such as object shape, are learned implicitly, thus edges can be effectively distinguished. Extensive experiments demonstrate that IRHED achieves state-of-the-art results on the widely used BSDS500 dataset. We also show the benefit of structured edge map for higher-level task, such as object proposal generation.