Probability Basics

Probability puts a number on uncertainty. A probability is a value between 0 (impossible) and 1 (certain) that says how likely something is. The most useful way to think of it is as a long-run frequency: how often the event would happen if you could repeat the situation over and over.

Probability as long-run frequency

Say a coin has a 0.5 chance of heads. That doesn't promise heads on any particular flip, or even 5 heads in 10. It's a statement about the long run: flip it enough times and the proportion of heads will close in on 0.5. This is the Law of Large Numbers, and it connects abstract probability to actual data. A related result comes later: the law says where the proportion settles, and the Central Limit Theorem describes the shape of its fluctuations along the way.

Early on, the proportion swings wildly, because a handful of flips tells you almost nothing. As the flips add up, the line settles close to the true value. (Even with P(heads) set to 0.5, the running proportion is usually slightly off 0.5. It converges toward 0.5 but never has to equal it exactly.)

Sample space and events

Before you apply any rule, list the possible outcomes. The sample space is every outcome the random phenomenon can produce, written S. Roll one die and S = {1, 2, 3, 4, 5, 6}. Flip two coins and S = {HH, HT, TH, TT}. That is four outcomes, not three, because HT and TH are different results that happen to give the same count of heads. An event is any collection of outcomes you care about. "Even" is the event {2, 4, 6}; "at least one head" is {HH, HT, TH}.

When every outcome in S is equally likely, probability becomes counting. Count the outcomes in the event, divide by the count in S, and that is the answer: P(even) = 3/6 = .5. Most of the work in an exam question is building S correctly, and most of the mistakes happen there too. Two coins have four outcomes, not three. Two dice have 36, not 21.

Four rules to start from

• Range. Every probability is between 0 and 1.
• Total. The probabilities of all the outcomes in S add to exactly 1.
• Complement. P(not A) = 1 − P(A). The chance of not rolling a 6 is 1 − 1/6 = 5/6.
• Addition for disjoint events. If two events cannot both happen, P(A or B) = P(A) + P(B). Rolling a 1 or a 2 is 1/6 + 1/6 = 1/3.

Events that cannot both happen are called disjoint, or mutually exclusive; the two words mean the same thing and both turn up in exam papers. The complement rule looks too simple to matter, but it is one of the most useful rules on this page, because a question about "at least one" is nearly always easier to answer from the other end. Work out the probability that it never happens, then subtract from 1. That step solves the first of the three classic problems below.

When events overlap: the general addition rule

The addition rule above has a condition: it fails as soon as two events can happen together. Adding P(A) and P(B) counts everything in the overlap twice, once inside each total, so you subtract the overlap once:

P(A or B) = P(A) + P(B) − P(A and B)

The rest of this lesson uses one example. A firm asked every one of its 200 employees how they get to work, then sorted the replies by whether the person is under 30.

Pick one employee at random. P(under 30 or cycles) is not .50 + .35 = .85, because the 48 people who are both have been counted in each of those totals. Subtracting them once leaves .50 + .35 − .24 = .61, and counting the same people directly agrees: 100 + 70 − 48 = 122 of 200, which is .61. For disjoint events the overlap is empty, so the shorter rule works for them and only for them.

Independence and the multiplication rule

Two events are independent when knowing that one happened does not change the probability of the other. Separate coin flips qualify. So do two draws from a deck, provided you put the first card back. For independent events, probabilities multiply:

P(A and B) = P(A) × P(B)

Two fair coins both landing heads is ½ × ½ = ¼. Three sixes in a row is (1/6)³ = 1/216. The rule extends to as many independent events as you like. Each extra event multiplies in another factor below 1, so a long streak is far rarer than a short one.

The addition and multiplication rules together explain why 7 is the most common total with two dice. Two dice have 36 equally likely outcomes, and the eleven possible totals do not share them evenly: one combination makes 2, and six make 7. Counting them all out gives a triangular distribution, which you can build a die at a time in the distribution playground and watch turn into a bell.

Conditional probability

Events that do affect each other need a new symbol. P(A | B), read "the probability of A given B", asks a narrower question than P(A). P(A) asks how common A is among everyone; P(A | B) asks how common it is among only the people who are also B. The definition divides one probability by another:

P(A | B) = P(A and B) / P(B)

Conditioning discards every row of the table that fails the condition, so the denominator shrinks to that row's own total rather than staying at 200. In the survey, P(cycles) = 70/200 = .35 across the whole company, while P(cycles | under 30) = 48/100 = .48. The gap between those two numbers is an association, and much of the rest of Stats 1 and Stats 2 is about testing associations like it. Independence is the special case where the gap is zero, so that P(A | B) = P(A) and learning B tells you nothing about A.

Rearranging the definition gives the general multiplication rule, which holds whether or not the events are independent:

P(A and B) = P(A) × P(B | A)

On the table, P(under 30 and cycles) = .50 × .48 = .24, matching the 48 of 200 you can read off directly. When the events are independent, P(B | A) equals P(B) and the general rule reduces to the short one. Cards dealt without replacement are the standard case where it does not: P(two aces) = (4/52) × (3/51) = 1/221, because the second probability depends on what the first draw removed.

Turning it around: Bayes' rule

P(A | B) and P(B | A) are different numbers, and confusing them is one of the most costly mistakes in applied probability. A screening test that catches 99% of real cases has told you P(positive | ill). The person holding the positive result wants P(ill | positive). Bayes' rule converts one into the other:

P(A | B) = P(B | A) × P(A) / P(B)

The numerator is the general multiplication rule read backwards. The denominator is the part that usually needs work, since P(B) is rarely given directly. It comes from the law of total probability: an event can be reached along several disjoint routes, and its overall probability is the sum of the probabilities of those routes.

P(B) = P(B | A) × P(A) + P(B | not A) × P(not A)

Positive results come from two routes: ill people the test catches, and healthy people it flags anyway. Together they make up every positive result. On the survey table, the same sum gives a number you can also read straight off it: P(cycles) = .48 × .50 + .22 × .50 = .35. Bayes' rule then answers the reversed question, P(under 30 | cycles) = .48 × .50 / .35 = .686, which matches the 48 of 70 cyclists you can count in the first column.

Written as one line, Bayes' rule looks like a formula to memorize. The tree below counts people instead, and counting makes the answer easier to see.

At the defaults the tree shows a good test, right 99% of the time in both directions, applied to a condition that affects 1 person in 1,000. Of 100,000 people it correctly flags 99 of the 100 who are ill. It also flags 999 of the 99,900 who are not, so barely 9% of the flagged group is ill. The test is working as designed. False positives outnumber true ones because there are a thousand times more healthy people than ill ones. So screening programs report positive predictive value (the share of positives who are ill) rather than accuracy.

Raise the prevalence and the same test's positive results become reliable: at 1 in 10, P(ill | positive) reaches 91.7%, and at 1 in 5 it reaches 96.1%. The test has not changed, only the population it is applied to. That is the argument for screening the people at risk rather than everybody. Bayesian reasoning goes much further than one 2 × 2 table, and Bayesian thinking in Stats 3 picks it up as a way of doing inference.

Three classic problems

Three practice problems use this lesson. Problem 5 reads all four kinds of probability off the survey table above, Problem 13 applies the general addition rule to a two-way count, and Problem 14 builds a screening tree from scratch.