In cooperative multi-agent exploration, the primary challenges typically include the unpredictability of state transitions and the variation among agents’ local knowledge and global information. To overcome these challenges, in this paper, we propose a decomposed action prediction technique for cooperative learning in the multi-agent reinforcement learning system. By parameterizing action spaces, where the agent must select both a discrete action and a set of continuous parameters for that action at each step, we propose partitioning action spaces into hierarchical role action spaces. This parameterization takes place according to the effects of action spaces on the environment and other agents. Through comparison with the most advanced methods of multi-agent algorithms in the StarCraft II micromanagement benchmark, we verify the originality of our methodology.