Logistic Regression

Logistic regression is a *classifier*, despite the name. It sorts points into two classes and reports how confident it is — not just a hard yes/no. It scores each point by which side of a line it falls on, then runs that score through the sigmoid to turn it into a probability.

From a score to a probability

First it computes a linear score z = w·x + b, exactly like linear regression. But a raw score can be any number, while a probability must live between 0 and 1. The sigmoid σ(z) = 1 / (1 + e^−z) squashes it: gentle near the middle, flattening toward 0 for very negative scores and 1 for very positive ones. A score of 0 maps to exactly 0.5.

z   = w·x + b          # signed distance-ish score
p   = 1 / (1 + e^-z)   # sigmoid → probability of class 1
pred = 1 if p >= 0.5 else 0

Training: minimize log-loss

We can't use squared error here — paired with the sigmoid it becomes non-convex and bumpy. Instead logistic regression minimizes log-loss (cross-entropy): tiny when a confident prediction is right, but exploding toward infinity when the model is *confidently wrong*. That asymmetry is what pushes the boundary into the right place.

loss = −[ y·log(p) + (1−y)·log(1−p) ]
# y=1, p→0  ⇒ loss → ∞   (confidently wrong, huge penalty)
# y=1, p→1  ⇒ loss → 0   (confident and correct)

Worked example

Suppose training settled on w = 2, b = −4, and a new point has x = 3. Then z = 2·3 − 4 = 2, so p = σ(2) ≈ 0.88 — predict class 1 with 88% confidence. A point at x = 2 gives z = 0, p = 0.5: it sits right on the boundary, a coin-flip.