Word Embeddings

Turn each word into a dense vector so that closeness in space means closeness in meaning — and relationships become directions.

9 min read Watch it move Build it

A word embedding turns each word into a vector — a list of numbers placing it as a point in space. The layout is learned so that geometry encodes meaning: words used in similar ways sit close together, and consistent relationships become consistent *directions*. Famously, king − man + woman lands near queen.

Why not one-hot?

A naive encoding gives each word its own slot — a huge sparse vector that is 1 in one position and 0 everywhere else. Every pair is then equally distant: cat is no closer to dog than to Tuesday. Embeddings are dense (a few hundred numbers) and *learned*, so similarity actually shows up as distance.

Learned from co-occurrence

The signal is co-occurrence — which words appear near each other in raw text. word2vec exploits this with a tiny prediction task over a sliding window. There are two flavors:

1Skip-gram — given the center word, predict its surrounding context words.
2CBOW (continuous bag of words) — given the context words, predict the center word.
3Either way, words that share contexts get pushed toward similar vectors — no labels required.

Meaning becomes arithmetic

Because the gender relationship is learned as a *fixed direction*, the step from man to king is roughly the same step as from woman to queen. That's why analogies can be solved with vector addition and subtraction.

# closeness is measured by COSINE similarity (angle), not raw distance
cos(a, b) = dot(a, b) / (norm(a) * norm(b))   # 1 = same direction

# the classic analogy
vec = embed["king"] - embed["man"] + embed["woman"]
nearest(vec)   # -> "queen"

Static vs contextual

Classic word2vec / GloVe embeddings are static — bank gets one vector whether it's a river or a vault. Modern transformers produce contextual embeddings, where the vector changes with the sentence. But the static-embedding intuition is the foundation underneath.

OperationTimeSpace

Lookup · vocabulary V × dimension d (often 100–300)O(1)O(V · d)

Check yourself

What real signal does word2vec use to learn that 'cat' and 'dog' are similar, with no labels?