Convolutional neural networks (CNN) have been excellent for scene classification in nature scene. However, directly using the pre-trained deep models on the aerial image is not proper, because of the spatial scale variability and rotation variability of the HSR remote sensing images. In this paper, a bidirectional adaptive feature fusion strategy is investigated to deal with the remote sensing scene classification. The deep learning feature and the SIFT feature are fused together to get a discriminative image presentation. The fused feature can not only describe the scenes effectively by employing deep learning feature but also overcome the scale and rotation variability with the usage of the SIFT feature. By fusing both SIFT feature and global CNN feature, our method achieves state-of-the-art scene classification performance on the UCM and the AID datasets. © Springer Nature Singapore Pte Ltd. 2017.