In this paper, we demonstrate the fabrication of refractive silicon microlens array with a large focal number and almost perfect spherical lens shape. Through a modified thermal reflow, photoresist microlens array of a large focal number is fabricated, which is then transferred into the silicon substrate by ion beam milling. To reach an accurate spherical lens profile, we both theoretically and experimentally study the practical factors that harm the pattern transfer fidelity, which mainly include the etching selectivity and faceting effect. Other secondary etching effects, such as the trenching effect and re-deposition effect, are also discussed. Based on these studies, a silicon microlens array with a focal number of 1.35 has been successfully obtained, with the profile error controlled well-below 0.121 mu m, less than lambda/6 within the whole infrared wavelength band. Besides, the fabricated microlens array exhibits a good uniformity and fine surface smoothness. The fabricated silicon microlens arrays can be applied in minatured infrared and terahertz imaging devices, or used as the master mould for soft lithography.