The MARL (Multi-Agent Reinforcement Learning) training framework for CGF (Computer Generative Force) behavior decision-making provides RL interaction and iteration for engagement-level models in military field. To ensure compatibility with various RL algorithms and meet the requirements of high-fidelity military simulations, We present a RL training framework which supports training engagement-level models in military field, and enables faster-than-real-time iteration. We also propose the Graph Normalized MAPPO(GNMAPPO) to enhance the adaptability of agents with local observations by integrating relationship information into their learning processes and successfully apply it to an unmanned aerial vehicle (UAV) reconnaissance mission. The experimental results demonstrate that our framework supports simultaneous interaction among multiple heterogeneous entities and GNMAPPO significantly outperforms baseline algorithm in UAV reconnaissance mission.