How a Search Engine Works

A search engine can't read the whole web every time you type a query — that would take forever. Instead it does the heavy lifting *in advance*: a crawler discovers pages, an inverted index files every word against the pages that use it, and a ranking step decides which matches matter most. Your query is then just a fast lookup plus a sort.

1. Crawling — discover the pages

A crawler (also called a bot or spider) starts from a set of known pages, downloads each one, and follows its links to new pages, repeating outward across the web. It respects robots.txt (which pages a site allows) and revisits pages to catch changes. The output is a huge pile of fetched page content.

2. Indexing — the inverted index

Scanning every crawled page at query time would be hopeless, so the engine builds an inverted index: the reverse of a page listing its words — a table mapping *each word* to the list of pages that contain it (its postings). Now answering 'which pages have this word?' is a single row lookup. A multi-word query intersects the postings lists to find pages containing *all* the terms.

inverted index (word -> pages that contain it):
  "cache"    -> [ p3, p7, p9 ]
  "browser"  -> [ p1, p3, p8 ]

query: "cache browser"
  intersect postings -> [ p3 ]     # p3 has BOTH words
  these are the candidates, still unordered

3. Ranking — PageRank and relevance

The candidates come back unordered, so the engine scores them. Relevance asks how well a page matches the query's words (term frequency, where the words appear, freshness). Importance is captured by PageRank: a link is a vote of confidence, and a page is important when *many important pages* link to it — a recursive definition, with a damping factor (~0.85) modelling a surfer who usually follows links but sometimes jumps at random. The final rank combines relevance and importance (modern engines blend in hundreds of signals, including learned ones).

4. The results page (SERP)

The top-scored pages are formatted into the search engine results page (SERP): the ranked list of blue links with titles and snippets, plus features like answer boxes. From your keystroke to that list, the engine only *looked up* an index row and *sorted* — all the crawling and indexing already happened, offline.