Conditional probability and Bayes, without the panic

One definition, and the rest is algebra

Conditional probability has a single definition: the probability of A given B is the probability of both divided by the probability of B. Everything else on this page - the multiplication rule, the law of total probability, Bayes' theorem - is that definition rearranged, and seeing it that way removes most of the memorisation.

The interpretation matters as much as the algebra. Conditioning on B means discarding every outcome where B did not happen and rescaling what is left so it sums to one. The denominator is the rescaling. Candidates who hold that picture rarely put the wrong quantity underneath.

Rearranged once, the definition gives the multiplication rule: the probability of both A and B is the probability of B times the probability of A given B. It is also the probability of A times the probability of B given A, and being able to write it either way is what makes Bayes' theorem a one-line derivation rather than a formula to remember.

The law of total probability

When the sample space is carved into disjoint pieces that cover everything, the probability of any event is the sum over those pieces of the probability of the piece times the conditional probability of the event given that piece. It is a weighted average, and the weights are the piece probabilities.

This is the workhorse. Nearly every Bayes question is really a total-probability question with one extra division at the end, and the total-probability step is where the arithmetic actually lives. Getting it right is most of the mark.

A useful discipline is to compute the denominator first and write it down as its own quantity before dividing anything by it. It is the probability of the evidence, it is reused for every posterior, and computing it once prevents the most common arithmetic slip in the topic.

A worked book of business

An insurer classifies its policyholders as standard, preferred or substandard. Sixty percent are standard, thirty percent preferred and ten percent substandard. The probability that a policyholder files a claim in a year is 0.02 for a standard risk, 0.01 for a preferred risk and 0.09 for a substandard one. A claim arrives. What is the probability it came from a substandard policyholder?

The denominator first. The unconditional probability of a claim is 0.60 times 0.02, plus 0.30 times 0.01, plus 0.10 times 0.09 - that is 0.012 plus 0.003 plus 0.009, which is 0.024. Then each posterior is its own contribution divided by that total.

So the posteriors are 0.012 over 0.024, or 0.500, for standard; 0.003 over 0.024, or 0.125, for preferred; and 0.009 over 0.024, or 0.375, for substandard. They sum to one, which is the check. The substandard group is a tenth of the book and more than a third of its claims, which is the whole point of the calculation.

Change the numbers and the shape stays the same. Three machines produce half, three-tenths and two-tenths of the output, with defect rates of 0.01, 0.03 and 0.06. The probability that a random item is defective is 0.026, and given a defect the probability it came from the third machine is 0.012 over 0.026, or 0.4615 - against a prior of only 0.20. Evidence moves probability toward whichever cause explains it best, and by how much is exactly what the arithmetic reports.

• Prior probabilities: 0.60 standard, 0.30 preferred, 0.10 substandard.
• Likelihoods: 0.02, 0.01 and 0.09 respectively.
• Probability of a claim: 0.024.
• Posteriors given a claim: 0.500, 0.125 and 0.375.

Bayes' theorem is the last division

Written out, Bayes' theorem says the posterior probability of a cause given the evidence equals the prior times the likelihood, divided by the total probability of the evidence. Every symbol in it has already appeared above; the theorem simply names the arrangement.

The reason it feels harder than it is comes from language rather than mathematics. Questions describe the likelihood - the probability of the evidence given the cause - and ask for the posterior, the probability of the cause given the evidence. Confusing the two is the single most common error in the topic, and it is a reading error, not a computational one.

Independence is a claim about numbers

Two events are independent when the probability of both equals the product of the two probabilities, which is the same as saying that conditioning on one does not change the other. It is a numerical condition, not a description of how the events feel, and it must be checked rather than assumed.

Suppose the probability of A is 0.35, of B is 0.45, and of both is 0.15. The product is 0.1575, which is not 0.15, so the two are not independent - and the probability of A given B is 0.15 divided by 0.45, or one third, against an unconditional 0.35. The events are close to independent and not independent, and on an exam that difference is the whole question.

The traps worth naming

Most lost marks in this topic come from a small, repeatable list. Working through it once, deliberately, is worth more than another twenty routine questions.

• Inverting the conditioning: computing the probability of the evidence given the cause when the question asked for the cause given the evidence.
• Forgetting that the pieces must be disjoint and must cover the whole sample space before the law of total probability applies.
• Assuming independence because two events sound unrelated, rather than checking that the joint probability equals the product.
• Confusing disjoint with independent. Two disjoint events with positive probability are strongly dependent: if one happens, the other cannot.
• Failing to take the complement on an at-least-one question, where one minus the probability of none is almost always faster than summing cases.
• Losing the check. Posteriors over a complete set of causes must sum to one, and that sum costs five seconds to verify.