In order to reduce the optical intensity of backscattering lights of the Cassegrainian optical antenna in the optical system of the laser communication terminal and improve the optical isolation of laser communication system, this paper proposes a method for designing high isolation level Cassegrainian optical antenna by increasing the characteristic value of the optical intensity of backscattering lights E/N. The method aims to improve the optical antenna isolation through reducing the optical intensity of backscattering lights along the incident optical path based on analyzing the theoretical model of the optical intensity of backscattering lights BRDF. A Cassegrainian optical antenna for laser communication is designed in this paper, and the method is used to optimize the isolation of this optical antenna by custom operands in ZEMAX. Then using TracePro software builds the optical structure model of this optical antenna before and after optimization, simulate the optical intensity of backscattering lights of each model, calculate and analyze the isolation of each model by using simulation data respectively. It is show that the backscattering rate of the system decreases from 0.0011956 to 0.00007515, and the isolation decreases from-29.224dB to-41.24dB. Optimization has improved the isolation 12.017dB of the optical system through use the method. © 2019 COPYRIGHT SPIE. Downloading of the abstract is permitted for personal use only.