Chapter 12 · OpenStax Ch 12
Linear Regression and Correlation
Fitting a line, reading r, and knowing when prediction is safe.
12.1 Least squares
Given points \((x_i, y_i)\), a straight line \(\hat{y} = a + bx\) predicts a \(y\) for every \(x\). Each point misses the line by its residual, \(y_i - \hat{y}_i\). The demo draws every residual as a square whose area is the squared miss, and the sum of squared errors is their total,
\[\mathrm{SSE} = \sum (y_i - \hat{y}_i)^2.\]
The least-squares line is the one line that makes this total as small as possible. Its slope and intercept follow from the data alone,
\[b = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}, \qquad a = \bar{y} - b\bar{x},\]
so it always passes through the point of means \((\bar{x}, \bar{y})\). Move points and watch the line follow. Then switch modes and try to beat it. Your orange squares can match the blue ones, never undercut them.
Reading: OpenStax Introductory Statistics, Sections 12.1 to 12.3.
Try it
- Move points is selected. Drag any of the nine points and the best line refits.
- Each residual is drawn as a real square. Best SSE is their total area.
- Press Your line vs best line. Two handles appear on a second line that you control.
- Drag the handles, or focus one and use the arrow keys. Your SSE changes and Yours above best by says how far you are from the minimum.
- Try to beat the best line. You cannot, which is what least squares means.
12.2 Correlation
The correlation coefficient measures how tightly points hug a straight line,
\[r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}},\]
always between \(-1\) and \(1\). Points above-right or below-left of the mean cross push \(r\) up, while the other two quadrants pull it down. Its square, the coefficient of determination \(r^2\), is the share of the variation in \(y\) that the line explains. Three cautions. The two “strong” presets have the same \(r\) with very different slopes, because \(r\) measures tightness rather than steepness. The curved preset follows a perfect pattern yet gives \(r \approx 0\), because \(r\) sees only straight-line association. And no value of \(r\) says that \(x\) causes \(y\).
Reading: OpenStax Introductory Statistics, Sections 12.2, 12.4.
Try it
- Strong, steep is selected. Read r, r² and Slope b.
- Press Strong, shallow. The slope changes but r does not. Correlation ignores the scale.
- Press Weak, then None, then Negative. Watch r move from near 0 to below 0.
- Press Curved, r ≈ 0. The dots clearly follow a curve, yet r is near 0. r measures straight lines only.
- Filled dots pull r up and hollow dots pull it down. The cross marks the two means.
12.3 The line is an estimate
The line you fit comes from one sample, and another sample would give another line. The grey cloud is a whole population of \((x, y)\) pairs with its own line, drawn dashed in orange. Each draw picks \(n\) of those points, fits them, and leaves the fitted line behind faintly. After a few dozen draws the lines form a band. Its width is the sampling variability of the slope, the same idea as the sampling distribution of a mean in Chapter 7. The spread of the slopes shrinks roughly like \(1/\sqrt{n}\), so raising \(n\) tightens the band around the population line. A slope printed to three decimals is still an estimate with a margin of error.
Reading: OpenStax Introductory Statistics, Section 12.4.
Try it
- The 400 dots are the population and the dark line is its true fit. Sample size, n is 15.
- Press Draw a sample. Fifteen dots are picked, fitted, and their line stays as a faint trace.
- Press Draw 25. The traces form a band. SD of the slopes measures its width.
- Slide n to 100 and press Draw 25 again. The band hugs the true line.
- New population starts over. Reset clears the traces but keeps the population.
12.4 Prediction and outliers
A fitted line predicts \(\hat{y} = a + bx\) for any \(x\) you like, but the data only vouch for the range they cover. Inside that range the prediction is an interpolation. Outside it is an extrapolation, shaded here as a caution zone, where the relationship may bend, level off, or stop making sense. Slide the prediction point past the last observed \(x\) and the status line changes its tone. The orange point is an outlier waiting to happen. Near the other points it barely matters. Dragged far to the right it gains leverage. One point tilts the whole line and drags \(r\) with it. Compare the solid line with the dashed one fitted without it, and always look for such points before trusting a slope.
Reading: OpenStax Introductory Statistics, Sections 12.5, 12.6.
Try it
- The diamond is a new point. The solid line includes it and the dashed line leaves it out.
- Drag the diamond, or focus it and use the arrow keys. Slope with the new point changes while Slope without it stays.
- Press Send the new point far right. One far point swings the solid line. That is leverage.
- Press Bring it back. Slide Predict y at x across the data and read Predicted y.
- Slide x past the data's range. The shaded zone says the prediction is an extrapolation, so trust it less.