Inference

Inference is the act of using a trained model to produce output, as opposed to training it. When you send a prompt, the model runs inference: it processes your input tokens, then generates output tokens one at a time, each requiring a pass through the whole network. Training happens once (or occasionally); inference happens every time anyone uses the model.

For businesses, inference is where the ongoing cost lives. API prices are inference prices, charged per input and output token. Running your own model means paying for the GPUs or servers that perform inference. Speed is measured as time to first token and tokens per second, both of which depend on model size, hardware and how busy the service is.

Many techniques exist to make inference cheaper and faster: smaller or distilled models, quantization, prompt caching, batching many requests together, and speculative decoding, where a small model drafts tokens that a large model checks. Reasoning models change the picture by spending extra inference on thinking before answering, trading cost and time for better results on hard problems.

Example

A company's chatbot handles 50,000 conversations a day. Training the underlying model cost the provider millions, once. The company pays for inference on every conversation, so it routes simple questions to a smaller, cheaper model and only sends complex ones to the largest model.

Common mistake

Budgeting only for visible output. Inference cost includes every input token too: system prompts, conversation history, retrieved documents and, for reasoning models, hidden thinking tokens. These often dwarf the visible reply.