Multi-agent reinforcement learning (MARL) has made significant advances in multi-agent systems. However, it is hard to learn a stable policy in complicated and changeable environment. To address these issues, a two-level attention network is proposed, which is composed of across-group observation attention network (AGONet) and intentional communication network (ICN). AGONet is designed to distinguish the different semantic meanings of observations (including friend group, foe group, and object/entity group) and extract different underlying information of different groups with across-group attention. Based AGONet, the proposed network framework is invariant to the number of agents existing in the system, which can be applied in large-scale multi-agent systems. Furthermore, to enhance the cooperation of the agents in the same group, ICN is used to aggregate the intentions of neighbors in the same group, which are extracted by AGONet. It obtains the understanding and intentions of their neighbors in the same group and enlarges the receptive filed of the agent. The simulation results demonstrate that the agents can learn complicated cooperative and competitive strategies and our method is superiority to existing methods.