Distillation
Also called knowledge distillation, model distillation, distilled model
Training a smaller AI model to imitate a larger one, so you get similar results for less cost.
Distillation is a training technique where a large "teacher" model's outputs are used to train a smaller "student" model. The student learns to reproduce the teacher's answers, and sometimes its full probability distributions, across many examples. The result is a model much cheaper and faster to run that keeps much of the teacher's ability, especially on the tasks it was distilled for.
Many small models released by AI labs are distilled from their larger siblings, and reasoning abilities have been successfully distilled into small open-weights models. Businesses use distillation too: they run a large model on many examples of their task, collect the outputs, and fine-tune a small model on them for high-volume use.
Distillation has limits. The student rarely matches the teacher on everything, and general knowledge suffers more than narrow skills. Provider terms of service often restrict using their model outputs to train competing models, so check the rules before distilling from a commercial API.
Example
A company uses a large model to label 50,000 product reviews by topic and sentiment. It fine-tunes a small model on those labels. The small model matches the large one on 94% of reviews and costs a tiny fraction as much to run on the next 10 million.
Common mistake
Distilling without an evaluation set. A student model can look fine on typical examples and fail badly on rare ones the teacher handled well. Compare teacher and student on a held-out set that includes edge cases before switching.