Multi-agent cooperation is one of the attractive aspects in multi-agent systems. However, during the process of cooperation, communication among agents is limited by the distance or the bandwidth. Besides, the agents move around and their neighbors appear or vanish, which makes the agents hard to capture temporal dependences and to learn a stable policy. To address these issues, a Spatial-Temporal Graph Attentional Long Short-Term Memory (LSTM) Scheme (STGA-LSTM), which is composed of spatial capture network and spatiotemporal LSTM network, is proposed. The spatial capture network is designed based on graph attention network to enlarge the agents’ communication range and capture the spatial structure of the multi-agent system. Based on the standard LSTM, a spatiotemporal LSTM network, which is in combination with graph convolutional network and attention mechanism, is designed to capture the temporal evolutionary patterns while keeping the spatial structure learned by spatial capture network. The results of simulations including mixed cooperative and competitive tasks indicate that the agents can learn stable and complicated strategies with STGALSTM.