Chapter 9 · OpenStax Ch 9
Hypothesis Testing (One Sample)
Null hypotheses, error types, and what a p-value really measures.
9.1 Hypotheses and error types
A hypothesis test starts with a claim to doubt, the null hypothesis \(H_0\). Here it claims the mean is 50. The alternative \(H_1\) is what we would conclude instead. We draw a sample, and if its mean lands past a decision cutoff we reject \(H_0\). Two things can go wrong. Rejecting a true \(H_0\) is a Type I error, and its probability is \(\alpha\), the orange tail. Failing to reject when the effect is real is a Type II error, probability \(\beta\), the blue tail. The test's power is \(1 - \beta\).
Drag the cutoff. Lowering \(\alpha\) always raises \(\beta\). The two errors trade off, and choosing \(\alpha = 0.05\) is a convention rather than a law. Only two things shrink both tails at once. A larger sample narrows both curves, and a larger real effect pulls them apart.
Reading: OpenStax Introductory Statistics, Sections 9.1, 9.2.
Try it
- The left curve is H₀ and the dashed right curve is the real effect of 5. The cutoff sits between them.
- Drag the cutoff right, or focus it and use the arrow keys. α falls and β rises.
- Press Cutoff for α = 5 %. The cutoff lands where the H₀ tail beyond it is exactly 0.05.
- Slide Sample size, n to 60. Both curves narrow, so they overlap less and Power rises.
- Slide True effect to 1. The curves nearly coincide and power collapses toward α.
9.2 The p-value
The p-value answers one question. If \(H_0\) were true, how often would a sample mean land at least this far from 50? It is the shaded tail area,
\[p = 2\,P\!\left(Z \ge |z|\right), \qquad z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}.\]
A small p-value says the data would be surprising under \(H_0\). It does not say how likely \(H_0\) is. Run a hundred tests with \(H_0\) true and look at the pile of p-values. It is flat. Small p-values are exactly as common as large ones, and one test in twenty crosses 0.05 by chance alone. Now make \(H_0\) false. The pile crowds toward zero, and the share below 0.05 is the power of the test.
Reading: OpenStax Introductory Statistics, Sections 9.3, 9.4.
Try it
- H₀ is true is selected, so samples come from μ = 50. Sample size, n is 25.
- Press Draw a sample. The shaded tails beyond x̄ are its p-value.
- Press Run 100 tests a few times. Share with p < 0.05 settles near 5%. Those are false alarms.
- Press H₀ is false and run again. The share below 0.05 is now the power.
- Slide Sample size, n up or the effect up and rerun. Power rises with both.
9.3 The full test, honestly run
Real tests rarely know \(\sigma\), so they use \(s\)1 and the t distribution. The steps never change. State \(H_0\) and \(H_1\). Draw the sample. Compute the test statistic \(t = (\bar{x} - \mu_0)/(s/\sqrt{n})\) with \(df = n - 1\). Find the p-value, compare it with \(\alpha\), and decide. The panel runs every step on a fresh sample each time you press the button.
Read the decision line carefully. “Fail to reject” means the data are compatible with \(H_0\), not that \(H_0\) is true. A small sample fails to reject almost everything. Set the true mean to 55 and run a hundred tests. Many fail to reject, and each of those is a missed real difference. Set it back to 50 and about \(\alpha\) of the tests reject anyway. Neither is a flaw in the method. It is what \(\alpha\) and \(\beta\) mean.
1 \(s\) divides by \(n - 1\), as everywhere on this site. Reading: OpenStax Introductory Statistics, Section 9.5.
Try it
- True population mean is 50, the same as μ₀. The test never sees it.
- Press Run one test. The five steps fill in, ending in a verdict and whether it was right.
- Press Run 100 tests twice. Rejection rate settles near α, the Type I rate.
- Slide the true mean to 55 and rerun. The rejection rate is now the power.
- Press α = 0.01. The cutoff moves out, fewer tests reject, and both rates fall.
What a p-value is not
Transcript
What a p-value is not.
A sample of 25. Its mean is 54.5. Test μ₀ = 50 with σ = 10 known. z = 2.27, p = 0.023.
p is the probability that H₀ is true. No.
p is the probability the result is due to chance. No.
1 − p is the probability of a real effect. No.
The one correct sentence.
If H₀ were true, a result at least this extreme would occur about 2 times in 100.
p describes the data, assuming H₀. It never describes H₀, given the data.