Alignment methods for large language models frequently fail to preserve safety under distribution shifts owing to the coupling of task objectives and safety preferences. We introduce Guardrail Guided Policy Optimisation (GGPO), a technique that decouples task rewards from safety constraints by inferring them separately from positive and negative demonstrations. This approach builds upon inverse constrained reinforcement learning, where constraints are inferred from expert behaviour, but extends it with an adversarial mechanism inspired by adversarial inverse reinforcement learning to jointly model rewards and costs as discriminators for goal-oriented and unsafe actions. The functional forms for the reward and cost are selected based on empirical factors promoting training stability. Policy optimisation is then conducted through a Lagrangian relaxation that imposes the inferred constraints, with algorithmic stability derived from two-timescale stochastic approximation principles. In a series of text-based navigation environments featuring moving hazards, GGPO eliminates constraint violations entirely for a transformer-based agent while adapting to safety requirements after distribution shifts. These findings indicate that inferring explicit, decoupled cost functions within a theoretically grounded framework enhances safety in settings requiring generalisation and safety adaptation.