Advancements in large language models (LLMs) have expanded their applications while raising concerns about safety and ethics, with post-training alignment emerging as a significant development to guide their behavior. In this paper, we introduce refined Direct Preference Optimization (rDPO), a method for improving the behavioral alignment of LLMs without the need for human-annotated data. The method involves creating synthetic data using self-critique prompting by a teacher LLM and then utilizing a generalized DPO loss function to distil to a student LLM. The loss function incorporates an additional external reward model to improve the quality of synthetic data, making rDPO robust to potential noise in the synthetic dataset. rDPO is shown to be effective in a diverse set of behavioral alignment tasks, such as improved safety, robustness against role-playing, and reduced sycophancy. Our results confirm that rDPO is more data-efficient than vanilla DPO, the current technique for aligning LLMs with offline data. Code available at github.com/vicgalle/refined-dpo