Deflectometry, with its noticeable advantages such as simple structure, large dynamic range, and high accuracy comparable to interferometry, has been one of the powerful metrological techniques for optical surfaces in recent years. In the "null" deflectometric transmitted wavefront testing of refractive optics, ray tracing of the test system model is required, in which both the miscalibration of system geometrical parameters and optical tolerances on tested optics could introduce significant geometrical aberrations in the testing results. In this paper, the geometrical aberration introduced by a system modeling error in the transmitted wavefront testing is discussed. Besides, a calibration method based on polynomial optimization of geometrical aberration is presented for the geometrical aberration calibration. Both simulation and experiment have been performed to validate the feasibility of the proposed calibration method. The proposed method can calibrate the optical tolerances on tested optics effectively, and it is feasible even with a large geometric error, providing a viable way to address the uncertainty in system modeling in transmitted wavefront testing of freeform refractive optics with large dynamic range. © 2021 Optical Society of America