Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Abstract

We introduce Direct Preference Optimization (DPO), a stable, performant, and computationally lightweight alternative to RLHF. DPO directly optimizes for human preferences without explicit reward modeling.

Key Contributions

• →DPO algorithm for preference optimization
• →Elimination of explicit reward model
• →Stable and efficient alignment training

Topics