RLHF and preference data, explained for teams that collect it

Most of what makes a modern assistant feel helpful happens after pre-training. Post-training teaches a model which of many plausible answers people actually want. The raw material for that step is preference data: humans comparing outputs and saying which is better, and why.

This guide explains how the main methods use that data and what it means for the people who produce it.

Where RLHF came from

The core idea predates large language models. In 2017, Christiano and colleagues showed that agents could learn complex behaviours from human judgements about which of two short clips was better, rather than from a hand-written reward function ^[1]. Their agents needed human feedback on less than one percent of their interactions with the environment ^[1]. The lesson that carried forward: people are far better at comparing outcomes than at specifying them.

The InstructGPT recipe

OpenAI's InstructGPT work set the template most teams still recognise ^[2]:

1. Demonstrations. Labelers write examples of the desired behaviour, and the model is fine-tuned on them with supervised learning.
2. Comparisons. Labelers rank several model outputs for the same prompt.
3. Reinforcement learning. A reward model trained on those rankings guides further fine-tuning.

The headline result is still striking. In human evaluations, outputs from a 1.3B parameter InstructGPT model were preferred to those of the 175B GPT-3, despite having roughly 100 times fewer parameters ^[2]. Better data and better objectives beat raw scale.

Anthropic's work on helpful and harmless assistants applied the same approach and collected preference data specifically for both helpfulness and harmlessness, studying how the two interact ^[3].

DPO: same data, simpler pipeline

Direct Preference Optimization, introduced by Rafailov and colleagues in 2023, showed that the reward model can be folded into the policy itself, so the standard RLHF objective can be optimised with a simple classification loss on preference pairs ^[4]. That removes the separate reward model and the reinforcement learning loop.

DPO changes the engineering, not the inputs. It still needs pairs of responses with a reliable judgement about which is better. If anything, it makes data quality more visible, because there is no reward model in between to absorb noise.

What good preference data looks like

The methods differ, but the data requirements are shared:

• Comparisons a domain expert would agree with. A preference on a chemistry answer is only useful if the person judging it knows chemistry.
• A rationale for each judgement. Short written reasons let reviewers audit decisions and catch rater drift.
• Calibrated guidelines. Raters need worked examples of edge cases: two good answers, two bad answers, an answer that is correct but unsafe.
• Agreement measured, not assumed. Overlap a share of items across raters and track agreement over time.
• Coverage, not just volume. Prompts should span the capabilities and failure modes the model will meet in use.

Common failure modes

Who should do the comparing

For general conversational quality, trained generalists can do excellent work. For code, maths, science, law and medicine, the comparison is a technical judgement, and it needs someone who can verify the answer rather than admire it. That is why we match preference work to specialists and keep a senior reviewer on borderline items.

Read more about how we vet experts, or see how our Train practice packages SFT, RLHF and preference data.