LLM-as-a-judge
Also called llm judge, ai grader, model-graded evaluation
Using an AI model to grade or compare AI outputs against criteria, instead of having people review every one.
LLM-as-a-judge means using a language model to evaluate outputs: scoring a response against a rubric, checking whether an answer is supported by a source, or picking the better of two responses. It makes evaluation fast and cheap enough to run on thousands of examples, which human review can not match.
A good judge prompt is specific. It gives clear criteria, a defined scale with descriptions of each score, and asks for a short justification before the score. Pairwise comparison ("which of these two answers is better, and why?") is often more reliable than absolute scores. Asking for structured output makes the results easy to aggregate.
Model judges have known biases. They can favour longer answers, answers in their own style, the first option shown, or confident tone. They also share blind spots with the models they judge. Good practice is to check judge scores against human ratings on a sample, swap the order of options in comparisons, and use several criteria rather than one overall score.
Example
A team tests two prompts for summarising articles. A judge model reads each article and both summaries, then rates each summary from 1 to 5 on accuracy, completeness and length, with a reason. Across 300 articles, prompt B scores higher on accuracy, which human spot checks confirm.
Common mistake
Trusting a judge you have not checked. If the judge's scores do not match what careful humans think on a sample of cases, the whole evaluation is measuring the wrong thing. Calibrate first, then scale.