Chapter 15 · Advanced Concepts
Bayesian Inference
Updating belief with data, through Bayes’ theorem, likelihood, and posteriors.
15.1 Bayes’ theorem
A test is right 99 % of the time when the disease is present and 95 % of the time when it is not. You test positive. Bayes’ theorem turns that around,
\[P(D \mid +) = \frac{P(+ \mid D)\,P(D)}{P(+ \mid D)\,P(D) + P(+ \mid \text{no } D)\,P(\text{no } D)}.\]
Count the thousand dots. Ten have the disease and about ten of those test positive. The other 990 are healthy, and 5 % of them test positive too, which is fifty people. So sixty test positive and only ten are ill. The answer is one in six, not 99 %. The prevalence is doing the work. A small error rate applied to a huge group beats a large catch rate applied to a tiny one. Drag the prevalence up and the same test becomes trustworthy.
Reading: this section is beyond the OpenStax text. See any introduction to conditional probability and base rates.
15.2 Likelihood
Until now a probability answered “given \(p\), how likely are these data?” Turn it around and hold the data fixed. The likelihood asks how well each candidate \(p\) explains the flips you actually got,
\[L(p) = \binom{n}{k} p^{k} (1-p)^{n-k}.\]
Same formula, different variable. The flips stay on screen while the slider moves, because the data are the one thing you are not free to change. The curve peaks at \(\hat{p} = k/n\), the maximum likelihood estimate. A likelihood is not a probability for \(p\), and the curve does not enclose an area of one. Only the ratios mean anything. A p that is ten times less likely explains your flips ten times worse.
Reading: this section is beyond the OpenStax text. See any introduction to maximum likelihood estimation.
15.3 Prior to posterior
Start with a belief about \(p\), see data, end with a better belief. Written as densities that is
\[\text{posterior} \propto \text{prior} \times \text{likelihood}.\]
Take a Beta prior with shapes \(a\) and \(b\) and Bernoulli data. The multiplication then has a closed form. The posterior is \(\text{Beta}(a + k, b + n - k)\) for \(k\) successes in \(n\) trials. Every observation adds one to a shape, so the prior behaves like data you already had. Beta(2, 2) is worth about two flips and is overturned quickly. Beta(20, 20) is worth forty and holds its ground. Watch the solid curve leave the dashed one and climb toward the true \(p\), narrowing as it goes.
Reading: this section is beyond the OpenStax text. See any introduction to Bayesian updating with conjugate priors.