Hyperspectral Images often suffer from missing pixels due to acquisition system problem. Missing pixels in images are usually tackled by means of interpolation methods by using neighborhood known pixels, but once the missing region is large, there may not be sufficient information in the neighborhood to reconstruct well. Fortunately, hyperspectral sensors are also often flown with boresighted, higher resolution panchromatic sensors. We propose a novel hyperspectral data recovery algorithm based on multi-source image fusion, guided by the gradient field of coincident panchromatic image in inpainting processing. The experiment results are illustrated with practical examples and verify the efficacy of this algorithm.