Bootstrap & Resampling
Back in Stats 1, the sampling distribution required imagining thousands of repeated studies you never actually ran. The bootstrap is a beautifully sneaky shortcut: treat the one sample you do have as a stand-in for the population, and resample from it, over and over, to see how much your statistic would wobble. No formulas, no normality assumption, just resampling and counting.
The trick: resample with replacement
Take your sample of n values. Draw a new sample of n values from it with replacement, so some original points get picked two or three times, others not at all. Compute your statistic (say, the mean) on that resample. Repeat thousands of times. The spread of those resampled statistics estimates the real sampling variability, directly from your data.
🎮 Build a Bootstrap Distribution
Top: your one sample of 15 values. Each resample picks 15 of them with replacement (taller orange stacks = picked more than once; faded dots = not picked). Its mean drops into the distribution below.
① Your sample (the current resample in orange)
② Bootstrap distribution of the mean, with the 95% percentile interval
What you get for free
- The bootstrap standard error is just the standard deviation of all those resampled means, an estimate of how much your statistic varies from sample to sample.
- The percentile confidence interval is even simpler: sort the bootstrap statistics and read off the 2.5th and 97.5th percentiles. That range is a 95% confidence interval — no t-tables required.
One refinement worth knowing about, since every package offers it. The plain percentile interval is slightly off when the bootstrap distribution is skewed or when the statistic is a biased estimate, because it assumes the resampling distribution is centered and shaped like the real one. The BCa interval (bias-corrected and accelerated) applies a correction for both and is the usual default recommendation; it costs nothing but a different argument. Percentile intervals are fine for a symmetric statistic like a mean, so the difference rarely shows up in a first course, but reach for BCa when you bootstrap a median, a ratio, or a correlation.
Why it's powerful: the bootstrap works for statistics that have no tidy formula: medians, correlations, ratios, the difference between two trimmed means, almost anything. When the classic formula doesn't exist or its assumptions don't hold, you can almost always bootstrap instead.
The one assumption that remains
The bootstrap leans on a single idea: that your sample is representative of the population. If the sample is biased or tiny, resampling it just faithfully reproduces that bias; the bootstrap can't conjure information that isn't there. It also can't rescue dependent data or wildly skewed tiny samples. But for a decent, representative sample, it's astonishingly reliable.
Why it matters: the bootstrap is one of the most important ideas in modern statistics: it trades mathematical derivation for raw computation, which computers have in abundance. It's the bridge between the formula-based methods of Stats 1–3 and the simulation-driven mindset of modern data analysis, and its sibling idea, resampling to see how a result holds up on data it hasn't seen, is the whole basis of cross-validation.
Common questions
How many bootstrap resamples do I need?
For standard errors, ~1,000 is plenty; for confidence intervals (which depend on the distribution's tails), 5,000–10,000 is the modern norm, and since computation is cheap there's no reason to skimp. Note what B does and doesn't fix: more resamples reduce simulation noise, but the information ceiling is set by your original n. B = 100,000 can't rescue a sample of 12.
My bootstrap interval and my t-interval disagree. Which one do I trust?
First check how far apart they are. For a mean from a reasonably symmetric sample the two should land within a whisker of each other, and a real disagreement is a signal rather than a nuisance: it usually means the sampling distribution is skewed, which is the assumption the t-interval makes and the bootstrap does not. In that case prefer the bootstrap, and prefer a BCa interval over a plain percentile one. Two things to rule out before concluding anything: too few resamples (an interval built on 1,000 replicates still jitters in its third digit, so rerun with 10,000 and see whether the disagreement survives) and a sample small enough that neither method is trustworthy. If the two agree, you have learned something too, which is that the parametric assumption was doing no harm here.
What is the difference between bootstrapping and permutation tests?
Different questions. The bootstrap resamples with replacement to estimate uncertainty: standard errors and confidence intervals for an estimate. A permutation test reshuffles group labels to build the null distribution — "what differences would chance produce if the labels meant nothing?" — yielding an exact p-value. Estimation → bootstrap; hypothesis testing → permutation.