This paper studies the moving object detection, i.e., analyzing the amount, position and size of the moving objects in instance-level, which is meaningful for many computer vision problems. However, the existing methods are still not satisfying in accuracy, portability and speed. In this paper, we propose a novel framework which detects moving objects by analysis the consistency of the moving foreground. Instead of directly performing cluster algorithms on the moving foregound, we take two stages: analyzing the composition according to the local density of the moving foreground points and locating the targets by regressing some anchors. In this way, the proposed method doesn’t need any training processes and can be efficiently performed to detect moving objects with arbitrary classes. Besides, we create our own publicly available dataset PDMOD with sufficient data, general challenges and convictive evaluation protocols to fill the scarcity of the evaluational datasets.