Cloud type classification based on ground-based cloud image observations is an important task in atmospheric research. Currently, two kinds of cloud image observations with infrared and visible light images are widely used for cloud classification. However, they are only independently analyzed and simply compared in the current study. The useful information from these two kinds of images is not fully utilized and integrated. The classification performance could be improved if taking full advantage of the complementary information of these two observations. Thus, first, a database containing these two kinds of cloud images with same temporal resolution is released in this study. Then, a two-observation joint encoding strategy of LBP (local binary pattern) features is proposed to implement cloud classification by encoding the joint distribution of LBP patterns in different observations, which captures the correlation between two observations. Experimental results based on this database show the significant superiority of the proposed method compared to the results based on the single observation.