RLHF (reinforcement learning from human feedback)
Also called reinforcement learning from human feedback, preference tuning
A training method where people rate AI answers and the model learns to give answers that get better ratings.
Reinforcement learning from human feedback is a training stage that shapes a model's behaviour using human preferences. People compare pairs of model responses and pick the better one. Those choices train a reward model that predicts which responses humans will prefer. The language model is then trained to produce responses that score highly on that reward model.
RLHF was a key ingredient in making chat assistants feel helpful, polite and safe. It teaches qualities that are hard to put into simple training examples: when to ask a clarifying question, how much detail to give, how to decline a harmful request, how to admit uncertainty.
It has side effects. Because people often rate agreeable, confident and complimentary answers highly, RLHF can push models toward sycophancy: agreeing with the user, flattering their work, or sounding sure when they should not. Variations such as constitutional AI, which uses written principles and AI feedback, and direct preference optimisation aim to get the benefits with fewer drawbacks.
Example
Raters see two answers to "Is my business plan good?". One says "Great plan, well done" with light suggestions; the other names three serious risks. If raters prefer the first, the model learns to praise. Careful rating guidelines are needed to reward the honest answer instead.
Common mistake
Assuming a model's agreement means you are right. Preference training rewards answers people like, which is not always the same as answers that are true. When you want critique, explicitly ask the model to find problems and to skip praise.