Quantization
Also called quantisation, quantized model, 4-bit model, gguf
Storing an AI model’s numbers with less precision so it uses less memory and runs faster, with a small loss in quality.
Quantization reduces the precision of a model's parameters. Models are usually trained with 16-bit or 32-bit numbers for each weight. Quantization converts those to 8-bit, 4-bit or even fewer bits. A 4-bit version of a model needs roughly a quarter of the memory of the 16-bit version, which can be the difference between running on a laptop and needing a data-centre GPU.
The trade-off is quality. Moderate quantization, such as 8-bit, usually has almost no noticeable effect. At 4-bit, well-made quantised models remain very usable, with small losses that show up more on complex reasoning, maths and less common languages. Below 4-bit, quality often drops sharply. Larger models tend to tolerate quantization better than small ones, so a quantised large model often beats a full-precision small model using the same memory.
When downloading open-weights models you will see formats and labels like GGUF, Q4_K_M, AWQ or int8. These describe the quantization method and level. Tools such as Ollama and LM Studio choose sensible defaults.
Example
A 14-billion-parameter model needs about 28 GB at 16-bit precision. A 4-bit quantised version needs about 8 to 9 GB, so it fits on a laptop with 16 GB of memory and runs at a usable speed.
Common mistake
Comparing models without noticing they are quantised differently. A model that seems weak locally might be a heavily quantised version of a strong one. When testing, note the quantization level, and try a higher-precision version if quality matters and memory allows.