A reward model is a function that scores how good an output is. That sounds trivial, but it turns out to be the load-bearing component of modern alignment: RLHF, best-of-n sampling, rejection sampling, and preference optimization all rest on having a model that can tell better outputs from worse ones. This post surveys the classical ways to build one — Bradley–Terry preference models, scalar regression, ranking models, and the more recent generative reward models — with an eye on what each one actually optimizes.

Throughout, write $x$ for a prompt (or context), $y$ for a response, and $r_\theta(x, y) \in \mathbb{R}$ for the scalar the reward model assigns to the pair. The whole game is choosing what data we fit $r_\theta$ to, and what loss we use.

Why not just regress "quality"?

The obvious idea is to collect absolute quality ratings and regress against them. The trouble is that humans are notoriously inconsistent at absolute scores — one annotator's 7/10 is another's 5/10, and the same annotator drifts over a day. Relative judgments ("is A better than B?") are far more stable and are the foundation of most reward models. So before the regression view, we start with the preference view.

Bradley–Terry: the preference model

The Bradley–Terry model (1952) is the workhorse. Given two responses to the same prompt, a "winner" $y_w$ and a "loser" $y_l$, it posits that the probability of preferring one is a sigmoid of the reward difference:

$$ P(y_w \succ y_l \mid x) = \sigma\!\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big) = \frac{1}{1 + e^{-(r_\theta(x,y_w) - r_\theta(x,y_l))}}. $$

We fit $r_\theta$ by maximum likelihood on a dataset of human comparisons, which gives the familiar pairwise logistic loss:

$$ \mathcal{L}_{\text{BT}}(\theta) = -\,\mathbb{E}_{(x,\,y_w,\,y_l)\sim\mathcal{D}} \Big[ \log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big) \Big]. $$

A few things are worth noticing. First, only reward differences are identified — the model is invariant to adding a constant to $r_\theta$, so the scale is arbitrary and needs to be pinned down (e.g. by normalization) if you care about absolute values. Second, this is exactly binary classification on the difference, so everything you know about logistic regression transfers. Third, in the standard RLHF recipe the reward model is a language model with the final unembedding replaced by a scalar head, trained with precisely this loss; the resulting $r_\theta$ is then frozen and used as the reward in PPO. This is the setup from InstructGPT and the line of work after it.

Scalar regression reward models

When you do have reliable numerical labels — say a 1–5 rating, an aggregated preference rate, or a verifiable score — you can regress directly. With a target $s(x,y)$ the loss is just mean squared error:

$$ \mathcal{L}_{\text{reg}}(\theta) = \mathbb{E}_{(x,y,s)\sim\mathcal{D}} \big[\big(r_\theta(x, y) - s\big)^2 \big]. $$

Regression is attractive when the signal is dense and calibrated — for instance a Likert scale collected under a tight rubric, or a continuous metric like an aesthetic score. It gives you an absolute, calibrated number rather than the arbitrary-scale output of Bradley–Terry. The catch is that it inherits all the annotator-inconsistency problems above, so in practice it is used where the target is genuinely well-defined. A common middle ground is to bin ratings and treat it as ordinal classification, or to convert ratings into pairwise comparisons and fall back to the BT loss.

Beyond pairs: ranking models

Preferences often come as full rankings of $K$ responses, not just pairs. The Plackett–Luce model generalizes Bradley–Terry to a ranking $y_1 \succ y_2 \succ \cdots \succ y_K$ as a product of "pick-the-best-of-the-rest" softmaxes:

$$ P(y_1 \succ \cdots \succ y_K \mid x) = \prod_{k=1}^{K} \frac{\exp\big(r_\theta(x, y_k)\big)}{\sum_{j=k}^{K} \exp\big(r_\theta(x, y_j)\big)}. $$

InstructGPT used a simple special case of this: from $K$ ranked responses it formed all $\binom{K}{2}$ pairs and averaged the BT loss over them, which both reuses the data efficiently and avoids overfitting from correlated comparisons within a batch.

Generative reward models

A scalar head throws away the thing language models are best at — generating text. Generative reward models (GRMs) instead keep the LM head and cast reward modeling as a generation problem. There are two broad flavors.

LLM-as-a-judge. Prompt a capable LM to compare two responses and emit a verdict. The "reward" is then the probability the judge assigns to the preferred answer,

$$ \mathcal{L}_{\text{GRM}}(\theta) = -\,\mathbb{E}_{(x,\,y_w,\,y_l)}\Big[\log p_\theta\big(\texttt{"A"} \mid x, y_w, y_l\big)\Big], $$

where "A" is the token naming the better response. Because the output is a token distribution, this is trained with the ordinary next-token cross-entropy — no separate scalar head, no separate loss.

Critique-then-judge. The more interesting variant has the model generate a written rationale before its verdict, so the reward is grounded in an explicit chain of reasoning. This is the reward-model analogue of chain-of-thought: it improves accuracy on hard comparisons and, as a bonus, makes the reward interpretable — you can read why an output scored low. Recent work pushes this further by training the critique itself with RL (letting the reward model "think" before scoring), which has become one of the more active directions.

The trade-offs are clear. Scalar BT models are cheap to evaluate (one forward pass, one number) and easy to plug into RL, but they are opaque and brittle. Generative reward models are slower and need careful decoding, but they are more accurate on hard cases, more interpretable, and they scale with inference-time compute — you can spend more tokens of reasoning to get a better judgment.

The problem that outlives every method: reward hacking

Whatever the parameterization, a reward model is a proxy, and optimizing a proxy hard enough breaks it — Goodhart's law. A policy trained against $r_\theta$ will find and exploit the regions where $r_\theta$ is wrong: verbosity, sycophancy, formatting tricks, or whatever spurious feature happened to correlate with reward in the training data. The standard defenses are all forms of restraint: a KL penalty keeping the policy close to the reference model,

$$ \max_\pi \; \mathbb{E}_{y\sim\pi}\big[r_\theta(x,y)\big] - \beta\, \mathrm{KL}\big(\pi(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big), $$

plus reward-model ensembling, early stopping, and periodically refreshing the reward model on fresh on-policy data. Underneath is a subtle point worth repeating: a reward model can be an excellent evaluator (high accuracy on held-out preferences) yet a poor optimization target, because optimization actively hunts for its blind spots. Robustness under optimization, not just accuracy on a test set, is the property we actually want — and the one we are worst at measuring.

Takeaways

  • Bradley–Terry turns pairwise preferences into a logistic loss on reward differences; it is the default and identifies reward only up to an additive constant.
  • Scalar regression fits absolute, calibrated scores with MSE — good when the target is genuinely well-defined, otherwise it inherits annotator noise.
  • Plackett–Luce extends preferences from pairs to full rankings; the common trick is to decompose a ranking into averaged pairwise BT losses.
  • Generative reward models recast scoring as generation, trained with ordinary cross-entropy; critique-then-judge trades compute for accuracy and interpretability.
  • Reward hacking is structural: the honest goal is a reward that is robust as a target, not merely accurate as an evaluator.