Attention (in AI models)
Also called attention mechanism, self-attention
The part of a transformer model that decides which other words matter most when processing each word.
Attention is the mechanism at the heart of transformer models. For each token, attention calculates how relevant every other token in the context is, then blends information from the most relevant ones. This lets the model build an understanding of each word that depends on its surroundings: "bank" next to "river" ends up represented differently from "bank" next to "loan".
Technically, each token produces three vectors: a query, a key and a value. Comparing one token's query with every other token's key gives attention scores. Those scores, normalised, weight a mix of the value vectors. Models run many of these calculations in parallel, called attention heads, which can learn to track different relationships such as grammar, references or topic.
Attention also explains some practical limits. The standard calculation compares every token with every other token, so cost grows quickly with context length. Much research goes into making attention cheaper for very long contexts. And models do not attend equally well to everything: details in the middle of very long inputs are sometimes used less than those near the start or end.
Example
When a model translates "She put the book on the table and then she read it", attention helps "it" connect strongly to "book" rather than "table", so the translation uses the right gender or word form in languages that need one.
Common mistake
Reading attention as an explanation of a model's reasoning. Attention weights show which tokens influenced others inside a layer, but the overall behaviour comes from many layers and heads interacting. They are a clue for researchers, not a reliable account of why a model gave an answer.