In areas including counterterrorism, security, diplomacy and supply chain optimisation, an analyst must make decisions under assumptions about the risks posed by an adversary. Research fields including operations research, decision theory, game theory, influence diagrams and adversarial risk analysis provide a rich variety of methods to model and solve such problems. Reinforcement learning (RL) is also an approach to sequential decision making that has been applied to specific problems involving risk. We propose multi-objective multi-agent RL (MOMARL) as a general-purpose approach to risk analysis. Using a MOMARL solver we model and solve variants of a problem in counterterrorism, including a notoriously difficult problem class: pessimistic bilevel optimisation under uncertainty.