Evals (AI evaluations)
Also called evaluations, llm evals, ai evaluation
Tests that measure how well an AI model or prompt performs on a set of example tasks.
Evals are systematic tests for AI systems. An eval has a set of inputs, an expected result or a way to judge results, and a score. Public benchmarks evaluate models in general. Private evals, built from your own real examples, tell you whether a model and prompt work for your specific use, which is what matters when building a product.
The simplest useful eval is a spreadsheet of 20 to 100 real inputs with notes on what a good answer looks like. Each time you change a prompt, switch models or adjust settings, you rerun the inputs and compare. Scoring can be exact matching (did it pick the right label?), code checks (is the JSON valid? did the tests pass?), human review, or another model acting as a judge.
Evals turn prompt work from guesswork into measurement. They catch regressions, where a change fixes one case and breaks three others, and they make model upgrades safe, because you can test a new model on your own tasks in an hour instead of trusting its marketing.
Example
A team classifying support tickets keeps 200 labelled tickets as an eval set. A new prompt raises accuracy from 86% to 91% but drops accuracy on billing tickets from 95% to 80%. The breakdown by category catches a problem an average alone would hide.
Common mistake
Testing only easy, typical cases. Evals are most valuable when they include the hard ones: ambiguous inputs, edge cases, past failures and adversarial examples. Every time the system fails in real use, add that case to the eval set.