Autoencoders

Train a network to copy its input to its output through a narrow bottleneck, forcing it to learn the data's essential structure.

8 min read Watch it move Build it

An autoencoder is a neural network trained to do something that sounds pointless: copy its input to its output. The catch is the middle. The encoder squeezes the input down into a small latent code; the decoder rebuilds the input from that code alone. Because the code is far smaller than the input, the network can't just memorize — it has to learn what's *essential*.

Encoder, bottleneck, decoder

1Encoder — maps the input x down to a compact latent code z (the bottleneck).
2Bottleneck — the narrowest layer, with far fewer numbers than the input. This is the squeeze that forces compression.
3Decoder — expands z back out to a reconstruction x̂, aiming to match the original x.

z      = encoder(x)         # compress: e.g. 784 numbers -> 32
x_hat  = decoder(z)         # rebuild back to 784

# reconstruction error — the only thing we train on
loss = mean((x - x_hat) ** 2)     # MSE; no labels needed

The target is the input itself

Training is unsupervised: the label for x is just x. There's no human annotation — the network learns features purely by being forced to reconstruct data through a bottleneck. Whatever survives the squeeze is the data's structure.

A worked example — denoising

Feed a corrupted image in (add random noise) but score the reconstruction against the clean original. To minimize error the network must learn to throw the noise away and keep the underlying digit, edge, or shape. The result is a denoising autoencoder — the same copy-through-a-keyhole idea, now cleaning data as a side effect.

Not a general-purpose compressor

An autoencoder is data-specific: one trained on faces compresses faces well and everything else badly. It will not beat a general codec like JPEG on arbitrary images. Its real value is *learned features*, denoising, and anomaly detection — inputs that reconstruct poorly are flagged as unusual.

OperationTimeSpace

Reconstruction quality · tighter bottleneck = more compression, more loss—code dim < input dim

Check yourself

Why does the bottleneck force an autoencoder to learn useful structure instead of just memorizing?