Probability Basics

Probability is how we put a number on uncertainty. A probability is just a value between 0 (impossible) and 1 (certain) that says how likely something is. The most useful way to think about it: a probability is the long-run frequency of an event if you could repeat the situation over and over.

Probability as long-run frequency

Say a coin has a 0.5 chance of heads. That doesn't promise heads on any particular flip, or even 5 heads in 10. It's a statement about the long run: flip it enough times and the proportion of heads will close in on 0.5. This is the Law of Large Numbers, and it's the bridge between abstract probability and actual data. It has a close relative you meet later: the law says where the proportion settles, while the Central Limit Theorem describes the shape of the wobble on the way there.

Two things to watch for: early on, the proportion swings wildly, because a handful of flips tells you almost nothing. As the flips pile up, the line stops wandering and hugs the true value. (And note: even a "fair" coin spends most of its time slightly off 0.5. It converges, but it never has to land exactly on it.)

Sample space and events

Before any rule can be applied, the situation needs a list. The sample space is every outcome the random phenomenon can produce, written S. Roll one die and S = {1, 2, 3, 4, 5, 6}. Flip two coins and S = {HH, HT, TH, TT}, which is four outcomes rather than three, because HT and TH are genuinely different results that happen to give the same count of heads. An event is any collection of outcomes you care about. "Even" is the event {2, 4, 6}; "at least one head" is {HH, HT, TH}.

When every outcome in S is equally likely, probability becomes counting. Count the outcomes in the event, divide by the count in S, and that is the answer: P(even) = 3/6 = .5. Most of the work in an exam question is building S correctly, and most of the mistakes live there too. Two coins have four outcomes, not three. Two dice have 36, not 21.

Four rules to start from

• Range. Every probability lies between 0 and 1. Nothing is less likely than impossible.
• Total. The probabilities of all the outcomes in S add to exactly 1.
• Complement. P(not A) = 1 − P(A). The chance of not rolling a 6 is 1 − 1/6 = 5/6.
• Addition for disjoint events. If two events cannot both happen, P(A or B) = P(A) + P(B). Rolling a 1 or a 2 is 1/6 + 1/6 = 1/3.

Events that cannot both happen are called disjoint, or mutually exclusive; the two words mean the same thing and both turn up in exam papers. The complement rule looks too small to matter and does more work than any other line on this page, because a question about "at least one" is nearly always easier to answer from the other end. Count the ways it fails to happen, then subtract from 1. That single move solves the first of the three classic problems below.

When events overlap: the general addition rule

The addition rule above carries a condition, and it breaks the moment two events can happen together. Adding P(A) and P(B) counts everything in the overlap twice, once inside each total, so the repair is to hand one copy back:

P(A or B) = P(A) + P(B) − P(A and B)

One example carries the rest of this lesson. A firm asked every one of its 200 employees how they get to work, then sorted the replies by whether the person is under 30.

Pick one employee at random. P(under 30 or cycles) is not .50 + .35 = .85, because the 48 people who are both have been counted in each of those totals. Subtracting them once leaves .50 + .35 − .24 = .61, and counting the same people directly agrees: 100 + 70 − 48 = 122 of 200, which is .61. Disjoint events are simply the case where the overlap is empty, which is why the shorter rule works there and only there.

Independence and the multiplication rule

Two events are independent when knowing that one happened leaves the probability of the other exactly where it was. Separate coin flips qualify. So do two draws from a deck, provided you put the first card back. For independent events, probabilities multiply:

P(A and B) = P(A) × P(B)

Two fair coins both landing heads is ½ × ½ = ¼. Three sixes in a row is (1/6)³ = 1/216. The rule extends to as many independent events as you like, and it is the reason a long run looks miraculous while a short one is unremarkable.

The addition and multiplication rules together settle an argument that breaks out at most games tables. Two dice have 36 equally likely outcomes, and the eleven possible totals do not share them evenly: one combination makes 2, and six make 7. Counting them all out gives a triangular distribution, which you can build a die at a time in the distribution playground and watch turn into a bell.

Conditional probability

The rule for events that do affect each other needs a new symbol. P(A | B), read "the probability of A given B", asks a narrower question than P(A): not how common something is among everyone, but how common it is among the people who are also B. The definition rescales one probability by another:

P(A | B) = P(A and B) / P(B)

Conditioning discards every row of the table that fails the condition, so the denominator shrinks to that row's own total rather than staying at 200. In the survey, P(cycles) = 70/200 = .35 across the whole company, while P(cycles | under 30) = 48/100 = .48. The gap between those two numbers is the association that the rest of Stats 1 and Stats 2 exists to test. Independence is the special case where the gap is zero, so that P(A | B) = P(A) and learning B tells you nothing.

Rearranging the definition gives the general multiplication rule, which holds whether or not the events are independent:

P(A and B) = P(A) × P(B | A)

On the table, P(under 30 and cycles) = .50 × .48 = .24, matching the 48 of 200 you can read off directly. Where the events happen to be independent, P(B | A) is just P(B) and the general rule collapses into the short one. Cards dealt without replacement are the standard case where it does not collapse: P(two aces) = (4/52) × (3/51) = 1/221, because the second probability has to know what the first draw removed.

Turning it around: Bayes' rule

P(A | B) and P(B | A) are different numbers, and treating them as the same one is the most expensive mistake in applied probability. A screening test that catches 99% of real cases has told you P(positive | ill). The person holding the positive result wants P(ill | positive). Bayes' rule converts one into the other:

P(A | B) = P(B | A) × P(A) / P(B)

The numerator is the general multiplication rule read backwards. The denominator is the part that usually needs work, since P(B) is rarely handed to you. It comes from the law of total probability: an event can be reached along several disjoint routes, and its overall probability is the sum of those routes.

P(B) = P(B | A) × P(A) + P(B | not A) × P(not A)

Positive results arrive along two routes: ill people the test catches, and healthy people it flags anyway. Add the two and you have every positive result there is. The survey table shows the same sum recovering a number you could also have read straight off it: P(cycles) = .48 × .50 + .22 × .50 = .35. Bayes then answers the reversed question, P(under 30 | cycles) = .48 × .50 / .35 = .686, which is the 48 of 70 cyclists you can count in the first column.

Set out as one line, Bayes' rule looks like a formula to be memorized. Counting people instead makes the answer obvious, and that is all the tree below does.

At the defaults the tree holds a good test, right 99% of the time in both directions, applied to a condition that affects 1 person in 1,000. Of 100,000 people it correctly flags 99 of the 100 who are ill, and it also flags 999 of the 99,900 who are not, so barely 9% of the flagged group turns out to be ill. Nothing is wrong with the test. False positives outnumber true ones because there are a thousand times more healthy people available to produce them, and this is why screening programs report positive predictive value rather than accuracy.

Slide the prevalence up and the same test becomes trustworthy: at 1 in 10 it reaches 91.7%, and at 1 in 5 it reaches 96.1%. The test never changed. Only the population it was pointed at did, which is the argument for screening the people at risk rather than everybody. Bayesian reasoning goes much further than one 2 × 2 table, and Bayesian thinking in Stats 3 picks it up as a way of doing inference.

Three problems worth knowing

Three practice problems run on this lesson: Problem 5 reads all four kinds of probability off the survey table above, Problem 13 works the general addition rule on a two-way count, and Problem 14 builds a screening tree from scratch.