In this paper, we propose a robust detection and tracking strategy for autonomous aerial refueling of unmanned aerial vehicles. The proposed framework includes two modules: a faster deep-learning-based detector (DLD) and a more accurate reinforcement-learning-based tracker (RLT). In the detection stage, the DLD achieves faster speed by combining the efficient MoblieNet with the you only look once framework. In the tracking stage, RLT is proposed to obtain target's position accurately and fastly by performing hierarchically positioning and adjusting target bounding box according to the reinforcement learning. The precision of drogue object tracking is 98.7, which is obviously higher than the other comparison methods. The speed of our network can achieve 15 frames/s on GPU Titan X. The experimental results validate the effectiveness and robustness of the proposed framework.