<p>This study investigates the evolutionary mechanisms of communication emergence by using temporally extended communication games, which are implemented as a multi-agent reinforcement learning environment. Our experimental results provide three key insights: (1) Reward delay is an essential factor that affects the possibility of communication emergence. When the reward delay is large, (2) direct utilities of signaling actions that facilitate the ritualization process become necessary for establishing communication, and (3) the sender’s well-established policy to maintain signaling is more critical than the receiver’s pre-evolved reaction policy. These insights provide a better understanding of the role of direct utilities for signaling, which may stimulate new exploration of the evolutionary origins of communications in nature.</p>