In order to efficiently learn multi-agent collaborative adversarial policies in partially observable adversarial scenarios, we propose a two-stage adversarial policy reinforcement learning (TSAP-RL) method. First, the adversarial policy is learned by using multi-step offline interaction data obtained by an expert policy in the first stage. Then, the adversarial policy is learned by using an prioritized fictitious self-play method in the second stage. However, it is challenging for reinforcement learning algorithms to use both multi-step offline and online interaction data to learn adversarial policies. To address this problem, we present an adaptive policy improvement algorithm based on Box-Cox transformation, which can use both offline and online interaction data for policy improvement. Moreover, to evaluate the policy efficiently, we also design an efficient policy evaluation method by using offline or online multi-step interaction data to learn the action-value and state-value functions simultaneously. The MiaoSuan wargame simulation system is used to validate the effectiveness of TSAP-RL, with evaluation results demonstrating that it can use offline and online interaction data to efficiently learn policies that can beat Blue side’s knowledge-based policy.