Scratches as the major defects in precision optical components are caused inevitably in the manufacturing process, which is harmful to the whole optical system. Most scratches on the surface of optical components are weak scratches with low contrast and uneven distribution of gray scale, which poses a significant problem for inspection. In this article, an end-to-end weak scratch inspection method based on novel scratch-enhancement methods and convolutional neural network (CNN) is proposed for optical components. To enhance weak scratches, a local maximum index (LMI) module and a direction-sensitive convolution (DSC) module are proposed to generate multilevel-feature maps using prior knowledge about scratch. Different from previous works utilizing the raw dark-field image as network input, these multilevel features are used as the inputs of encoder-decoder module for training. After training, the whole inspection model can infer weak scratches from raw dark-field test images in an end-to-end manner. Experimental results show that the proposed model achieves pixel accuracy of 92.48% and IoU at 77.27% on the test data set. It outperforms the networks without adding prior knowledge, which shows that prior knowledge is much helpful for weak scratch inspection. Moreover, compared with other classical methods and CNN-based methods, the proposed method achieves the best performance in the weak scratch inspection.