No single memory is fast, large, and cheap at the same time. Fast memory (SRAM) is expensive and small; cheap memory (disk) is huge but slow. The fix is to *stack* several kinds in a hierarchy — tiny fast levels on top of big slow ones — and let the fast levels hold whatever is being used right now.
The levels, top to bottom
1Registers — a few dozen cells inside the CPU; the only place arithmetic actually happens. Sub-nanosecond.
2Cache (L1, L2, L3) — small SRAM holding recently used data. L1 is a few cycles away; L3 tens of cycles.
3Main memory (RAM) — large DRAM, roughly 100x slower than L1, measured in gigabytes.
4Disk (SSD / HDD) — the biggest, cheapest, slowest level, and the only one that survives a power-off.
Each step trades speed for size
Going *down* a level multiplies capacity and cost-efficiency but also multiplies latency. A rough ladder: register under 1 ns, L1 ~1 ns, RAM ~100 ns, SSD ~10-100 us, HDD ~10 ms. From cache to disk is about a *million-fold* jump in latency.
Why it works — locality
The hierarchy is a bet, and the bet is locality. Temporal locality: data used now is likely to be used again soon, so keep it near. Spatial locality: data near what you just touched is likely next, so move whole blocks up at once. Because real programs have both, the *average* access lands in a fast level even though the data really lives far down.
Worked example — average memory access time
The payoff is measured by AMAT (average memory access time): AMAT = hit time + miss rate * miss penalty. Suppose L1 answers in 1 cycle, misses 5% of the time, and a miss costs 100 cycles to fetch from RAM.
Even with a 100-cycle penalty, a 95% hit rate keeps the *average* near the speed of L1 — that is the whole point of the stack.
Hits matter more than raw speed
Halving the miss rate from 5% to 2.5% drops AMAT from 6 to 3.5 cycles. Improving the hit rate of a fast level usually beats making a slow level faster, because the slow level is reached rarely.
