In the piezoelectric sensor-based debris flow flume tests, the impact force coefficient (k) obtained from inversion has a significant dispersion range, which causes considerable uncertainty in the debris flow impact force. To reduce the dispersion range of k, this study designed a new measurement system by integrating the Fiber Bragg Grating (FBG) and the elastic cantilever beam as the core sensing unit. The impact of debris flows on the sensing unit causes the bending deformation of the cantilever beam, which subsequently causes the wavelength shift of the FBG fixed on the cantilever beam. The optical signals are converted into the dynamic strain signals of the cantilever beam using the photosensitive characteristic of the FBG. According to the mechanical characteristics of the cantilever beam and the Bingham model of debris flow, the method of inversing k using the strain signals was proposed. Based on the FBG sensor system, 45 tests were conducted under different working conditions. The inversion model obtained 38 sets of k by analyzing the test data. The standard deviation and dispersion coefficient of k were 0.4 and 0.25, respectively, making the k in the range of 0.9 to 2.8, which was superior to the k derived from the traditional measurement mode.