Attention Mechanism

For each item, decide which other items matter using query–key matches, then blend their values — the core operation inside transformers.

9 min read Watch it move Build it

Attention lets a model decide, for *each* item, which other items matter right now. Instead of treating all inputs equally, it dynamically pulls in exactly the context it needs. This single operation is the heart of every transformer.

Query, key, value

Every item produces three vectors, by analogy with looking something up:

1Query q — what the current item is *looking for* (its question).
2Key k — what each item *offers* (its label, to be matched against queries).
3Value v — the actual *content* an item contributes when attended to.

The four steps

1Score — compare the query to every key with a dot product: a bigger number means more aligned, so more relevant.
2Scale — divide the scores by √d_k to keep them from getting too large as the vectors grow.
3Softmax — turn the scores into attention weights: positive numbers that sum to 1, a focus distribution over the items.
4Blend — take the weighted sum of the values using those weights. That blended vector is the output — a custom summary of the sequence for this item.

# scaled dot-product attention (Vaswani et al., 2017)
scores  = Q @ K.T / sqrt(d_k)     # how well each query matches each key
weights = softmax(scores)         # rows sum to 1 — the focus distribution
output  = weights @ V             # weighted blend of the values

Worked feel

Reading "the animal didn't cross the street because it was tired," the query from it matches the key of animal more strongly than street. Softmax puts most weight on animal, so it's output vector is mostly the *value* of animal — the model has resolved the reference by attending to it.

Quadratic cost

Every item compares to every other, so attention is O(n²) in the sequence length n. Doubling the sequence quadruples the work and memory — the main reason long-context efficiency is an active research area.

OperationTimeSpace

Attention over a sequence · n items, dimension d; the n×n score matrix dominatesO(n² · d)O(n²)

Check yourself

What guarantees that an item's attention weights form a focus distribution summing to 1?