Hypothesis Testing (Two Samples)

10.1 Comparing two means

Two groups, two sample means, and one question. Is the gap between them bigger than chance alone would produce? The two-sample t-test divides the observed difference by its own noise,

\[t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{s_A^2/n_A + s_B^2/n_B}},\]

and asks how often a \(t\) this large would appear if the populations were really equal.¹ Drag the two means together and sample. The differences pile up around zero and most p-values are large. Pull the means apart and the pile shifts, but with \(n = 15\) many samples still fail to reach 0.05, because two overlapping populations produce overlapping samples. Raise \(n\) and the same gap becomes unmistakable. Overlap in the curves and significance in the test are different things.

¹ This is Welch's version, which does not assume equal spreads. Its degrees of freedom come from the Welch–Satterthwaite formula. \(s\) divides by \(n - 1\). Reading: OpenStax Introductory Statistics, Sections 10.1, 10.2.

10.2 Comparing two proportions

Two versions of a sign-up page are shown to \(n\) visitors each. Version A converts 10 % of them, version B 13 %. Is B really better, or did it get luckier visitors? The two-proportion z-test pools both samples into one overall rate \(\hat{p}_c\) and judges the gap against the noise that rate implies,

\[z = \frac{\hat{p}_A - \hat{p}_B}{\sqrt{\hat{p}_c (1 - \hat{p}_c)\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}}.\]

Run the test a hundred times with 200 visitors per version and look at the pile of p-values. The three-point gap is real, yet most runs miss it. Slide \(n\) to 2 000 and the pile crowds toward zero. Small real differences are detectable, but only by large experiments. That is why online tests run for weeks.

Reading: OpenStax Introductory Statistics, Section 10.3.

10.3 Paired samples

Twelve students take a test, get a week of coaching, and take it again. Students differ from each other far more than coaching changes any one of them. Treating the before and after scores as two unrelated groups buries a real improvement under that between-student spread.

The fix is to test the differences, one per subject, so \(d_i = \text{after}_i - \text{before}_i\). A paired t-test is just a one-sample t-test on those differences, \(t = \bar{d} / (s_d / \sqrt{n})\), and it never sees the between-student spread at all. Press “Treat as pairs”. The lines join each student to their own second score. The standard deviation collapses from the spread between students to the spread of the changes, and the p-value drops, for exactly the same numbers. Whenever data come in natural pairs, pair them.

Reading: OpenStax Introductory Statistics, Section 10.4.