Fine-grained visual classification (FGVC) remains challenging because a majority of samples have large intra-class variations and small inter-class variations. However, samples belonging to one category are essentially identical in some discriminative visual patterns. Intuitively, we want models to reinforce the relationship between these discriminative visual patterns and image-level labels. In this paper, we propose a feature comparison based channel attention (FCCA) to achieve this intuition. In FCCA, the feature comparison mechanism is designed to recognize discriminative visual patterns. The weights assignment scheme guarantees that feature channels related to discriminative visual patterns have larger weights. The state-of-the-art performance has been achieved on two public FGVC datasets. Extensive experiments further prove the effectiveness of our method.