Definition

RLHF (Reinforcement Learning from Human Feedback) is a training method in three moves: collect people’s judgments about which model responses they prefer, use those judgments to train a reward model that can score responses, then use reinforcement learning to push the language model toward higher scores.

The problem it addresses is that a “good answer” is hard to define with rules. Whether a response is helpful, honest, appropriately toned or risky is something people can compare at a glance, but hard to write down as a loss function. RLHF turns human judgment into a training signal, and it’s one of the key steps in taking a model from “good at continuing text” to “useful and reliable.”

How It Works

The classic RLHF pipeline has three stages:

Step 1: Supervised fine-tuning (SFT). People write a set of high-quality instruction/response examples, and the pretrained model is fine-tuned on them to learn basic instruction following. This model is the starting point for what follows.

Step 2: Train a reward model. For the same prompt, the model generates several responses. Annotators rank them or compare them in pairs. That comparison data trains a reward model that outputs a score for any prompt/response pair, reflecting human preference.

Step 3: Optimize with reinforcement learning. The model generates responses to many prompts, the reward model scores them, and an RL algorithm (commonly PPO, proximal policy optimization) updates the model to favor high-scoring responses. To stop the model from gaming the reward model or drifting into unnatural language, training usually adds a constraint that keeps it from moving too far from the starting model.

RLHF has limits:

  • Cost: it needs a lot of human annotation, and RL training is complex and can be unstable.
  • Reward hacking: the model may learn to exploit quirks of the reward model rather than truly improve, for example by getting wordier or more flattering.
  • Biased preferences: annotators don’t always agree, and their views can carry bias.

Researchers have proposed alternatives and refinements. DPO (direct preference optimization) skips the separate reward model and RL loop and optimizes the model directly on preference data, which simplifies the pipeline. Other approaches replace some human feedback with AI feedback (often called RLAIF), or have the model critique and revise itself against a written set of principles.

Typical Applications

  • Making chat responses better match user intent and more organized
  • Reducing harmful, offensive or dangerous output
  • Encouraging the model to admit uncertainty rather than make things up
  • Tuning response style, such as brevity and politeness
  • Optimizing on tasks like code and math using verifiable outcome signals
  • Fine-tuning: supervised fine-tuning is RLHF’s first step.
  • Large language model: RLHF’s main target.
  • Alignment: the research area concerned with making AI behavior match human intent and values; RLHF is a flagship method.
  • Hallucination: RLHF can reduce it somewhat but can’t eliminate it.

Papers: Training language models to follow instructions with human feedback, Direct Preference Optimization