Synthetic Aperture Radar (SAR) is widely used in terrain classification, object detection, and other fields. Compared with anchor-based detectors, anchor-free detectors remove the anchor mechanism and implement detection box encoding in a more elegant form. However, anchor-free detectors are limited by complex scenes caused by geometric transformations such as overlay, shadow, vertex displacement during SAR imaging. And the scattered power distribution of noise is similar to the edge of the object, making it difficult for the detector to locate the edge of the SAR object accurately. In order to alleviate these problems, we propose a high-speed and high-performance SAR image anchor-free detector. First, we propose a shallow feature refinement (SFR) module to effectively extract and retain the detailed information of objects while coping with deformed complex scenes. Second, we analyze the optimization focus of the detector at different training iterations and propose Iteration-Aware Loss to guide the detector, making the detector more accurately locate the edge of the object disturbed by the noise scattered power distribution. Third, number estimation helps to detect objects with more flexible criteria in box selection without manual labor. Compared with mainstream optical object detectors and SAR dedicated detectors, our method achieves the best speed-accuracy trade-off on the SAR-Ship dataset, with 96.4% Average Precision when the value of Intersection over Union (IoU) is 50% AP50 at 64.9 Frames Per Second (FPS). The experimental results prove the effectiveness of our method. Author