Mandibular prognathism is a facial skeletal malocclusion. Until now, the genetic mechanism has been unclear. The goal of this study was to identify candidate genes or genomic regions directly associated with mandibular prognathism development, by employing whole genome sequencing. A large Chinese family was recruited, composed of 9 affected and 12 unaffected individuals, and the inheritance pattern of this family tends to be autosomal dominant. A single-nucleotide missense mutation in the ADAMTS1 gene (c. 742I>T) was found to segregate in the family, given that the affected individuals must be heterozygous for the mutation. For mutation validation, we screened this candidate mutation and 15 tag single-nucleotide polymorphisms in the coding sequence of ADAMTS1 among 230 unrelated cases and 196 unrelated controls using Sequenom Massarray and found that 3 in 230 cases carried this mutation and none of the controls did. Final results suggested that 2 single-nucleotide polymorphisms (rs2738, rs229038) of ADAMTS1 were significantly associated with mandibular prognathism.