Chapter 9 · OpenStax Ch 9

Hypothesis Testing (One Sample)

Null hypotheses, error types, and what a p-value really measures.

9.1 Hypotheses and error types

A hypothesis test starts with a claim to doubt, the null hypothesis \(H_0\). Here it claims the mean is 50. The alternative \(H_1\) is what we would conclude instead. We draw a sample, and if its mean lands past a decision cutoff we reject \(H_0\). Two things can go wrong. Rejecting a true \(H_0\) is a Type I error, and its probability is \(\alpha\), the orange tail. Failing to reject when the effect is real is a Type II error, probability \(\beta\), the blue tail. The test's power is \(1 - \beta\).

Drag the cutoff. Lowering \(\alpha\) always raises \(\beta\). The two errors trade off, and choosing \(\alpha = 0.05\) is a convention rather than a law. Only two things shrink both tails at once. A larger sample narrows both curves, and a larger real effect pulls them apart.

Reading: OpenStax Introductory Statistics, Sections 9.1, 9.2.

Try it

  1. The left curve is H₀ and the dashed right curve is the real effect of 5. The cutoff sits between them.
  2. Drag the cutoff right, or focus it and use the arrow keys. α falls and β rises.
  3. Press Cutoff for α = 5 %. The cutoff lands where the H₀ tail beyond it is exactly 0.05.
  4. Slide Sample size, n to 60. Both curves narrow, so they overlap less and Power rises.
  5. Slide True effect to 1. The curves nearly coincide and power collapses toward α.

9.2 The p-value

The p-value answers one question. If \(H_0\) were true, how often would a sample mean land at least this far from 50? It is the shaded tail area,

\[p = 2\,P\!\left(Z \ge |z|\right), \qquad z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}.\]

A small p-value says the data would be surprising under \(H_0\). It does not say how likely \(H_0\) is. Run a hundred tests with \(H_0\) true and look at the pile of p-values. It is flat. Small p-values are exactly as common as large ones, and one test in twenty crosses 0.05 by chance alone. Now make \(H_0\) false. The pile crowds toward zero, and the share below 0.05 is the power of the test.

Reading: OpenStax Introductory Statistics, Sections 9.3, 9.4.

Try it

  1. H₀ is true is selected, so samples come from μ = 50. Sample size, n is 25.
  2. Press Draw a sample. The shaded tails beyond x̄ are its p-value.
  3. Press Run 100 tests a few times. Share with p < 0.05 settles near 5%. Those are false alarms.
  4. Press H₀ is false and run again. The share below 0.05 is now the power.
  5. Slide Sample size, n up or the effect up and rerun. Power rises with both.

9.3 The full test, honestly run

Real tests rarely know \(\sigma\), so they use \(s\)1 and the t distribution. The steps never change. State \(H_0\) and \(H_1\). Draw the sample. Compute the test statistic \(t = (\bar{x} - \mu_0)/(s/\sqrt{n})\) with \(df = n - 1\). Find the p-value, compare it with \(\alpha\), and decide. The panel runs every step on a fresh sample each time you press the button.

Read the decision line carefully. “Fail to reject” means the data are compatible with \(H_0\), not that \(H_0\) is true. A small sample fails to reject almost everything. Set the true mean to 55 and run a hundred tests. Many fail to reject, and each of those is a missed real difference. Set it back to 50 and about \(\alpha\) of the tests reject anyway. Neither is a flaw in the method. It is what \(\alpha\) and \(\beta\) mean.

1 \(s\) divides by \(n - 1\), as everywhere on this site. Reading: OpenStax Introductory Statistics, Section 9.5.

Try it

  1. True population mean is 50, the same as μ₀. The test never sees it.
  2. Press Run one test. The five steps fill in, ending in a verdict and whether it was right.
  3. Press Run 100 tests twice. Rejection rate settles near α, the Type I rate.
  4. Slide the true mean to 55 and rerun. The rejection rate is now the power.
  5. Press α = 0.01. The cutoff moves out, fewer tests reject, and both rates fall.

What a p-value is not

Transcript

What a p-value is not.

A sample of 25. Its mean is 54.5. Test μ₀ = 50 with σ = 10 known. z = 2.27, p = 0.023.

p is the probability that H₀ is true. No.

p is the probability the result is due to chance. No.

1 − p is the probability of a real effect. No.

The one correct sentence.

If H₀ were true, a result at least this extreme would occur about 2 times in 100.

p describes the data, assuming H₀. It never describes H₀, given the data.