Hypothesis Testing (One Sample)

9.1 Hypotheses and error types

A hypothesis test starts with a claim to doubt, the null hypothesis \(H_0\). Here it claims the mean is 50. The alternative \(H_1\) is what we would conclude instead. We draw a sample, and if its mean lands past a decision cutoff we reject \(H_0\). Two things can go wrong. Rejecting a true \(H_0\) is a Type I error, and its probability is \(\alpha\), the orange tail. Failing to reject when the effect is real is a Type II error, probability \(\beta\), the blue tail. The test's power is \(1 - \beta\).

Drag the cutoff. Lowering \(\alpha\) always raises \(\beta\). The two errors trade off, and choosing \(\alpha = 0.05\) is a convention rather than a law. Only two things shrink both tails at once. A larger sample narrows both curves, and a larger real effect pulls them apart.

Reading: OpenStax Introductory Statistics, Sections 9.1, 9.2.

9.2 The p-value

The p-value answers one question. If \(H_0\) were true, how often would a sample mean land at least this far from 50? It is the shaded tail area,

\[p = 2\,P\!\left(Z \ge |z|\right), \qquad z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}.\]

A small p-value says the data would be surprising under \(H_0\). It does not say how likely \(H_0\) is. Run a hundred tests with \(H_0\) true and look at the pile of p-values. It is flat. Small p-values are exactly as common as large ones, and one test in twenty crosses 0.05 by chance alone. Now make \(H_0\) false. The pile crowds toward zero, and the share below 0.05 is the power of the test.

Reading: OpenStax Introductory Statistics, Sections 9.3, 9.4.

9.3 The full test, honestly run

Real tests rarely know \(\sigma\), so they use \(s\)¹ and the t distribution. The steps never change. State \(H_0\) and \(H_1\). Draw the sample. Compute the test statistic \(t = (\bar{x} - \mu_0)/(s/\sqrt{n})\) with \(df = n - 1\). Find the p-value, compare it with \(\alpha\), and decide. The panel runs every step on a fresh sample each time you press the button.

Read the decision line carefully. “Fail to reject” means the data are compatible with \(H_0\), not that \(H_0\) is true. A small sample fails to reject almost everything. Set the true mean to 55 and run a hundred tests. Many fail to reject, and each of those is a missed real difference. Set it back to 50 and about \(\alpha\) of the tests reject anyway. Neither is a flaw in the method. It is what \(\alpha\) and \(\beta\) mean.

¹ \(s\) divides by \(n - 1\), as everywhere on this site. Reading: OpenStax Introductory Statistics, Section 9.5.