Remarkable progress has been made in real-time semantic segmentation by leveraging lightweight backbone networks and auxiliary low-level training tasks. Despite several techniques have been proposed to mitigate accuracy degradation resulting from model reduction, challenging regions often exhibit substantial uncertainty values in segmentation results. To tackle this issue, we propose an effective structure named Uncertainty-aware Boundary Attention Network(UBANet). Specifically, we model the segmentation uncertainty via prediction variance during training and involve it as a regularization item into optimization objective to improve segmentation performance. Moreover, we employ uncertainty maps to investigate the role of low-level supervision in segmentation task. And we reveal that directly fusing high- and low-level features leads to the overshadowing of large-scale low-level features by the encompassing local contexts, thus hindering the synergy between the segmentation task and low-level tasks. To address this issue, we design a Low-level Guided Feature Fusion Module that avoids the direct fusion of high- and low-level features and instead employs low-level features as guidance for the fusion of multi-scale contexts. Extensive experiments demonstrate the efficiency and effectiveness of our proposed method by achieving the state-of-the-art latency-accuracy trade-off on Cityscapes and CamVid benchmark.