A novel convolution architecture PatchCNN is proposed for extending 2-D grid convolution to the nongrid structured data: point clouds, without any intermediate data representation. Previous studies implicitly capture local shape pattern from the meaningful subset or a local region without considering the interaction among points of the local region. The PointPatch module in our deep network, in spirit to the 8-pixels neighborhood in the 2-D image, explicitly models geometric relationship among points in the local region. We adopt a light 3-D convolution network to adaptively integrate features of the PointPatch module. The integrated features encode geometric relationship and the impact of surrounding points, which brings sufficient shape awareness and robustness for point cloud perception. Additionally, in our work, the convolution weight on each point is treated as a Lipschitz continuous function approximated by multilayer perceptron (MLP) and integrated features in the local region. Theoretically, the explicit learning strategy proposed in PatchCNN introduces inductive bias beneficial to the learning shape pattern in 3-D Euclidean space. Extensive experiments on ModelNet40 and ScanNet v2 data sets demonstrate that the proposed method achieves the competitive performance on par or even better than state-of-The-Art methods. © 2004-2012 IEEE.