Chapter 15 · Advanced Concepts
Bayesian Inference
Updating belief with data, through Bayes’ theorem, likelihood, and posteriors.
15.1 Bayes’ theorem
A test is right 99 % of the time when the disease is present and 95 % of the time when it is not. You test positive. Bayes’ theorem turns that around,
\[P(D \mid +) = \frac{P(+ \mid D)\,P(D)}{P(+ \mid D)\,P(D) + P(+ \mid \text{no } D)\,P(\text{no } D)}.\]
Count the thousand dots. Ten have the disease and about ten of those test positive. The other 990 are healthy, and 5 % of them test positive too, which is fifty people. So sixty test positive and only ten are ill. The answer is one in six, not 99 %. The prevalence is doing the work. A small error rate applied to a huge group beats a large catch rate applied to a tiny one. Drag the prevalence up and the same test becomes trustworthy.
Reading: this section is beyond the OpenStax text. See any introduction to conditional probability and base rates.
Try it
- 1 in 100 has the condition. The test catches 99% of cases and clears 95% of the healthy.
- The thousand dots are laid out in four blocks. Read Positives per 1 000 and …of whom really have it.
- Press Show only the positives. Only those people remain, and their share with the condition is P(has it | tested positive).
- Slide People who have it, per 100 down to 0.1. Almost every positive is now a false one.
- Slide Specificity to 100. False positives vanish and a positive result becomes trustworthy.
15.2 Likelihood
Until now a probability answered “given \(p\), how likely are these data?” Turn it around and hold the data fixed. The likelihood asks how well each candidate \(p\) explains the flips you actually got,
\[L(p) = \binom{n}{k} p^{k} (1-p)^{n-k}.\]
Same formula, different variable. The flips stay on screen while the slider moves, because the data are the one thing you are not free to change. The curve peaks at \(\hat{p} = k/n\), the maximum likelihood estimate. A likelihood is not a probability for \(p\), and the curve does not enclose an area of one. Only the ratios mean anything. A p that is ten times less likely explains your flips ten times worse.
Reading: this section is beyond the OpenStax text. See any introduction to maximum likelihood estimation.
Try it
- Ten flips are shown as filled and hollow dots. They stay fixed while you move Candidate p.
- Slide p from 0.50 toward Best-supported p. The marker climbs the curve to its peak.
- Read Times worse than the best as you slide away from the peak. Only such ratios mean anything.
- Press Snap to the best p. The marker jumps to k/n, the maximum likelihood estimate.
- Press New flips for new data, then slide Flips observed, n to 60. The curve narrows with more data.
15.3 Prior to posterior
Start with a belief about \(p\), see data, end with a better belief. Written as densities that is
\[\text{posterior} \propto \text{prior} \times \text{likelihood}.\]
Take a Beta prior with shapes \(a\) and \(b\) and Bernoulli data. The multiplication then has a closed form. The posterior is \(\text{Beta}(a + k, b + n - k)\) for \(k\) successes in \(n\) trials. Every observation adds one to a shape, so the prior behaves like data you already had. Beta(2, 2) is worth about two flips and is overturned quickly. Beta(20, 20) is worth forty and holds its ground. Watch the solid curve leave the dashed one and climb toward the true \(p\), narrowing as it goes.
Reading: this section is beyond the OpenStax text. See any introduction to Bayesian updating with conjugate priors.
Try it
- The prior is Beta(2, 2), the dashed grey curve. Prior mean reads 0.5.
- Press Observe one. One flip of the hidden coin moves a or b by exactly one and the solid posterior shifts.
- Press Observe 10 a few times. Posterior mean climbs toward The coin's true p, which is 0.7.
- Press Reset, slide Prior shape a and b both to 20, then observe again. The posterior moves much less.
- A strong prior takes more data to overturn. The Posterior readout names the shape parameters.