Don't just separate the classes — carve the widest empty street between them. Only the points touching the curb (support vectors) decide where the boundary goes.
A support vector machine (SVM) separates two classes by carving the widest possible empty street between them, rather than settling for just any dividing line. The line down the middle of the street is the boundary, and only the handful of points touching its edges — the support vectors — actually decide where it sits.
The maximum-margin idea
Many lines might separate two classes perfectly, but they're not equally good. A boundary that skims right past the data is fragile — a little noise flips a point to the wrong side. SVM instead maximizes the margin: the width of the empty street, measured to the nearest point of each class. A roomier gap generalizes better to unseen data.
Only the support vectors matter
The boundary is pinned in place by the few points sitting exactly on the edges of the street — the support vectors. Move any *other* point (as long as it stays off the street) and the boundary doesn't budge. The entire model is defined by a small subset of the data, which is what makes SVMs compact and robust.
The optimization
For a boundary w·x + b = 0, the margin width works out to 2 / ‖w‖. Maximizing the margin therefore means minimizing `‖w‖` while still classifying every point correctly (y·(w·x + b) >= 1). That's a convex quadratic program with a single global optimum — no local minima to worry about.
minimize (1/2)*||w||^2
subject to y_i * (w·x_i + b) >= 1 for every point i
margin width = 2 / ||w|| -> smaller ||w|| means a wider street
Soft margins and the kernel trick
1Soft margin. Real data overlaps, so a *hard* margin (zero violations) is often impossible. A penalty C allows a few points to sit inside the street or on the wrong side. Small C = a wider, more forgiving street; large C = fewer violations but a tighter fit.
2Kernel trick. When no straight line works, an SVM implicitly maps points into a higher-dimensional space where they *do* separate linearly — using a kernel like the RBF (Gaussian) kernel — without ever computing those coordinates explicitly. This lets a fundamentally linear method draw curved boundaries.
SVM vs logistic regression
Both draw a linear boundary, but they optimize different things. Logistic regression maximizes the *likelihood* of the labels and uses every point. An SVM maximizes the *geometric margin* and depends only on the support vectors — so it tends to be more robust to points far from the boundary, but it outputs a raw score, not a calibrated probability.
OperationTimeSpace
Train (kernel SVM) · n points; quadratic programO(n²·d) to O(n³)O(n²)
Predict · s = number of support vectorsO(s·d)O(s·d)
Check yourself
What determines where an SVM's decision boundary sits?
