Chapter 10 · OpenStax Ch 10

Hypothesis Testing (Two Samples)

Comparing two means or two proportions, and why pairing helps.

10.1 Comparing two means

Two groups, two sample means, and one question. Is the gap between them bigger than chance alone would produce? The two-sample t-test divides the observed difference by its own noise,

\[t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{s_A^2/n_A + s_B^2/n_B}},\]

and asks how often a \(t\) this large would appear if the populations were really equal.1 Drag the two means together and sample. The differences pile up around zero and most p-values are large. Pull the means apart and the pile shifts, but with \(n = 15\) many samples still fail to reach 0.05, because two overlapping populations produce overlapping samples. Raise \(n\) and the same gap becomes unmistakable. Overlap in the curves and significance in the test are different things.

1 This is Welch's version, which does not assume equal spreads. Its degrees of freedom come from the Welch–Satterthwaite formula. \(s\) divides by \(n - 1\). Reading: OpenStax Introductory Statistics, Sections 10.1, 10.2.

Try it

  1. Two curves show the groups. Drag either curve's mean marker to set the true difference.
  2. Press Sample both groups. Fresh samples are drawn and Welch's test gives a p-value.
  3. Put the means together and press Repeat 100 times. About 5% reject by chance.
  4. Pull the means 5 apart and repeat. Share with p < 0.05 is now the power.
  5. Slide Sample size per group, n to 60 and repeat. The pile of differences narrows and power rises.

10.2 Comparing two proportions

Two versions of a sign-up page are shown to \(n\) visitors each. Version A converts 10 % of them, version B 13 %. Is B really better, or did it get luckier visitors? The two-proportion z-test pools both samples into one overall rate \(\hat{p}_c\) and judges the gap against the noise that rate implies,

\[z = \frac{\hat{p}_A - \hat{p}_B}{\sqrt{\hat{p}_c (1 - \hat{p}_c)\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}}.\]

Run the test a hundred times with 200 visitors per version and look at the pile of p-values. The three-point gap is real, yet most runs miss it. Slide \(n\) to 2 000 and the pile crowds toward zero. Small real differences are detectable, but only by large experiments. That is why online tests run for weeks.

Reading: OpenStax Introductory Statistics, Section 10.3.

Try it

  1. Version A signs up 10% of visitors and version B 13%. Each gets 200 visitors.
  2. Press Run the test. Two fresh counts appear and the pooled z gives a p-value.
  3. Press Repeat 100 times. At n = 200 most tests miss the 3-point gap.
  4. Slide Visitors per version, n to 2000 and repeat. Now most tests find it.
  5. Set both rates equal and repeat. The share rejecting falls to about 5%.

10.3 Paired samples

Twelve students take a test, get a week of coaching, and take it again. Students differ from each other far more than coaching changes any one of them. Treating the before and after scores as two unrelated groups buries a real improvement under that between-student spread.

The fix is to test the differences, one per subject, so \(d_i = \text{after}_i - \text{before}_i\). A paired t-test is just a one-sample t-test on those differences, \(t = \bar{d} / (s_d / \sqrt{n})\), and it never sees the between-student spread at all. Press “Treat as pairs”. The lines join each student to their own second score. The standard deviation collapses from the spread between students to the spread of the changes, and the p-value drops, for exactly the same numbers. Whenever data come in natural pairs, pair them.

Reading: OpenStax Introductory Statistics, Section 10.4.

Try it

  1. Twelve subjects are measured before and after, with a true improvement of 3.
  2. Treat as two independent groups is selected. Read the p-value.
  3. Press Treat as pairs. The same data now give a smaller p-value, because SD between subjects no longer counts as noise.
  4. Press New subjects several times. The paired p is smaller most of the time.
  5. Slide True improvement per subject to 0. Now about 5% of redraws reject, either way.