Logistic Regression
So far the outcome y has always been a number. But often it's a yes/no: passed or failed, clicked or didn't, recovered or didn't. Fit a straight regression line to a 0/1 outcome and it cheerfully predicts probabilities like −0.3 or 1.4 — nonsense. Logistic regression fixes this with one elegant move.
Bend the line into an S
Instead of predicting y directly, logistic regression predicts the probability that y = 1, and runs the linear part through the logistic (sigmoid) function, which squashes any number into the range 0–1:
P(y=1) = 1 / (1 + e−(b₀ + b₁x))
The result is a smooth S-curve: near-0 for low x, near-1 for high x, with a steep transition in between. It can never escape 0 or 1, so the predictions always make sense as probabilities.
🎮 Fit the S-Curve
Dots at the top fired (y=1); dots at the bottom didn't (y=0). Slide the midpoint and steepness to fit the logistic curve — and notice the gray straight line shooting past 0 and 1 where it shouldn't.
Log-odds and odds ratios
Logistic regression earns its "regression" name honestly: it is linear — just on the log-odds scale. The model says the log-odds of the outcome is a straight line, b₀ + b₁x. That means the coefficient b₁ is a change in log-odds per unit of x, and exponentiating it gives the odds ratio:
- Odds ratio = eb₁: how the odds of the outcome multiply for each one-unit increase in x.
- An odds ratio of 1 means x has no effect; above 1 means it raises the odds; below 1 means it lowers them.
- Both lines of that translation, the log-odds model and OR = eb, are on the formula sheet, a row apart.
Odds ratios are the currency of medical and epidemiological papers, which makes them awkward to compare against the psychology literature's Cohen's d. The effect-size converter translates between the two when you need to put an odds ratio next to a standardized mean difference.
Probabilities aren't linear, but log-odds are. A one-unit change in x shifts the odds by a constant factor, but its effect on the probability depends on where you are on the curve — biggest in the steep middle, tiny out in the flat tails. That's why a single "effect on probability" number doesn't exist for logistic models.
Is the model any good?
Ordinary regression hands you R² and an F-test. Logistic regression has neither, so it substitutes two things that software prints under unfamiliar names. An omnibus test compares your model against an intercept-only model on the likelihood scale and reports a χ²: this is the logistic answer to "does the model beat knowing nothing?". A pseudo-R² (Nagelkerke, Cox & Snell, McFadden) rescales that same likelihood improvement onto a 0-to-1 range so it reads like something familiar. Treat it as a rough comparative number rather than variance explained, since the three flavors disagree with each other on the very same model.
For individual predictors most output shows a Wald test, which is fine for modest coefficients and unreliable for large ones. When a predictor matters enough to argue about, refit without it and compare the two models by likelihood ratio instead.
From probability to decision
To actually classify, you pick a threshold (often 0.5): predict "yes" when the curve is above it, "no" below. Move the threshold to trade false positives against false negatives — a choice driven by which mistake is costlier (missing a disease vs. a false alarm). Logistic regression is the workhorse behind credit scoring, medical risk models, and a great deal of machine-learning classification. Choosing that threshold well, and measuring what it costs you, is the subject of classification metrics.
Why it matters: binary outcomes are everywhere, and logistic regression is the standard, interpretable tool for them. It extends naturally to multiple predictors (numeric and categorical) exactly like ordinary regression — same linear model, new link function.
To fit one yourself, admissions.csv holds 160 applicants with GRE scores, GPAs and an admitted/rejected outcome, plus the coefficients to check your own against.
Want to feel the difference between an odds ratio and a probability change? Problem 25 of the practice problems moves the same two-unit step along the curve twice and gets .215 in one place and .017 in the other.
Common questions
How do I interpret an odds ratio?
It's the multiplier on the odds for each one-unit increase in the predictor. OR = 1.5 means each extra unit multiplies the odds of the outcome by 1.5 (+50%); OR = 0.8 shrinks them by 20%; OR = 1 means no effect. Two cautions: "odds" are p/(1−p), not probability — and an impressive-sounding OR can mean a tiny absolute change when the baseline risk is low.
Why is my odds ratio in the millions, with a confidence interval to match?
Your data is almost certainly separated: some predictor (or combination of them) splits the outcome perfectly, so every case above a cut-off is a 1 and every case below is a 0. No finite coefficient fits that best, because a steeper curve always fits it better than the last one. The estimate simply runs off toward infinity until the software gives up and prints wherever it stopped. Fitting eight perfectly split points by hand, the slope passes an odds ratio of 4 million by iteration 10 and the algorithm loses its footing entirely by iteration 20. It is a signal, not a failure: a predictor that good is usually a proxy for the outcome, a category with no cases in one cell, or a sample too small to have any overlap. The standard remedies are to merge the sparse categories, drop the offending predictor, or fit a penalized model (Firth logistic regression) that keeps the estimate finite.
What is the difference between odds and probability?
Probability is successes ÷ all attempts; odds are successes ÷ failures. A 75% probability is odds of 3 (three successes per failure). They diverge most in the middle of the range (p = .5 is odds = 1) and converge for rare events — p = .01 is odds ≈ .0101, which is why odds ratios approximate risk ratios only when the outcome is uncommon.