In resource-constrained environment, agents must balance task performance with resource utilization. This paper proposes a novel Resource-Constrained Reward-Delayed Multi-Agent Proximal Policy Optimization (RD-MAPPO) algorithm based on deep reinforcement learning for resource-constrained game. The framework of RD-MAPPO is specifically designed for multi-agent cooperative scenarios, incorporating a dynamic penalty mechanism that adapts to resource constraints through resource utilization and value acquisition metrics. Additionally, to mitigate the issue of reward sparsity, an extra probe reward function is introduced. Experiments in a CR3BP (Circular Restricted Three-Body Problem) scenario demonstrate that RD-MAPPO achieves superior performance over MAPPO and QMIX in both win rate and resource remaining rate across varying adversary ratios.