Hierarchical Model and X (HMAX) presents an invariant feature representation, following the mechanisms

of the visual cortex. Although HMAX in object recognition is robust, scale and shift invariant, it has been shown to be sensitive to rotational deformation. To address this, we propose a novel patch selection method saliency and keypoint based patch selection (SKPS). In addition, we suggest an SKPS based HMAX model (S-HMAX). In contrast to HMAX that employs the random patch deriving a significant amount of redundant information, S-HMAX uses SKPS to extract fewer numbers of features with better distinctiveness. To show the effectiveness of S-HMAX, we apply it to object categorization on TU Darmstadt (TUD) database. Experimental results demonstrate that the performance of S-HMAX is a significant improvement on that of conventional HMAX.