Training data

Training data is the material a model learns from. For large language models, pre-training data typically includes public web pages, books, academic papers, code repositories, and licensed or curated datasets, filtered for quality and deduplicated. It can total many trillions of tokens. Later training stages use smaller, carefully built datasets of instructions, conversations and human preference ratings.

Training data shapes everything about a model: what it knows, what languages it handles well, which writing styles feel natural to it, and which viewpoints and biases it absorbs. Topics that appear often and consistently are learned well. Rare topics, recent events and specialised knowledge that is not public are learned poorly or not at all.

Training data also raises ongoing questions about copyright, consent and privacy, which are being argued in courts and policy around the world. For users, two practical points matter: the model's knowledge ends at its cutoff date, and whether your own conversations are used for future training depends on the provider and your settings, which you can usually check and change.

Example

A model answers questions about popular programming languages fluently but struggles with an in-house language used at one company, because that language barely appeared, or never appeared, in its training data.

Common mistake

Assuming that anything on the internet is "in" the model and can be recalled exactly. Models learn patterns, not a searchable copy of the web. They often can not reproduce a specific page, and when they try, they may blend sources or invent details.