PEEK-based biomaterials have great potential applications as hard tissue substitutes in bone tissue engineering. However, inherent bio-inert properties limited their clinical use. In order to improve the bioactivity, in this work, zirconium ions were implanted into the carbon fiber-reinforced PEEK (CFR-PEEK) using plasma immersion ion implantation (PIII) technology. Surface morphologies and chemical compositions of Zr-PIII treated samples were analyzed by scanning electron microscopy (SEM), X-ray photoelectron spectroscopy (XPS), respectively. The results indicated that nanostructures and ZrO2 nanoparticles were formed on the surface of CFR-PEEK after Zr-PIII. Mechanical tests revealed that nanohardness, elastic modulus, and elastic resistance increased after implantation, especially for the elastic modulus with a maximum value of about 14 GPa, which is much close to that of human natural bone. In vitro cellular experiments showed that Zr-PIII treated samples enhanced the initial adhesion of rBMSCs, spreading and proliferation significantly. Moreover, the heightened ALP activity, collagen secretion, and extracellular matrix mineralization suggested that Zr-PIII treatment could greatly lead to an up-regulated osteogenic differentiation of rBMSCs on CFR-PEEK. In addition, antibacterial properties were also investigated and the results showed that Zr-PIII treated CFR-PEEK with nanostructures exhibited obvious antibacterial activity against S. aureus but no effect on E. coli.