AlgoPlus//machine learning / self-attention

Self-Attention & Multi-Head

Every token attends to every token; different heads capture different relations.

Head

Head 1

Query

—

Attention head

Legend

Attention weight

Token

AI Tutor Workspace

—

In a nutshell

Self-attention lets every word in a sentence look at every other word and decide how much to focus on each. A word asks a question (its query) and every word offers a label (its key); matching them gives focus weights that blend the words' contents (their values). Several heads do this in parallel, each catching a different kind of relationship.

Ready

Press play to begin the cinematic walkthrough.

Let every word interview every other word at once and write down how much it cares — an N×N grid of attention. Several interviewers (heads) ask about different things.

Key terms

Go deeper in the lesson

Read the full theory, intuition & complexity for Self-Attention & Multi-Head.