Scene parsing is an important task in computer vision and many issues still need to be solved. One problem is about the non-unified framework for predicting things and stuff and the other one refers to the inadequate description of contextual information. In this paper, we address these issues by proposing a Hierarchical Deep Probability Analysis(HDPA) method which particularly exploits the power of probabilistic graphical model and deep convolutional neural network on pixel-level scene parsing. To be specific, an input image is initially segmented and represented through a CNN framework under Gaussian pyramid. Then the graphical models are built under each scale and the labels are ultimately predicted by structural analysis. Three contributions are claimed: unified framework for scene labeling, hierarchical probabilistic graphical modeling and adequate contextual information consideration. Experiments on three benchmarks show that the proposed method outperforms the state-of-the-arts in scene parsing. © 2017 IEEE.