DPO vs PPO vs GRPO: Preference Tuning Compared

Three ways to turn preferences into weights, compared on the thing that decides between them: how many models must be resident and whether training needs online sampling.