Linear Regression and Correlation

12.1 Least squares

Given points \((x_i, y_i)\), a straight line \(\hat{y} = a + bx\) predicts a \(y\) for every \(x\). Each point misses the line by its residual, \(y_i - \hat{y}_i\). The demo draws every residual as a square whose area is the squared miss, and the sum of squared errors is their total,

\[\mathrm{SSE} = \sum (y_i - \hat{y}_i)^2.\]

The least-squares line is the one line that makes this total as small as possible. Its slope and intercept follow from the data alone,

\[b = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}, \qquad a = \bar{y} - b\bar{x},\]

so it always passes through the point of means \((\bar{x}, \bar{y})\). Move points and watch the line follow. Then switch modes and try to beat it. Your orange squares can match the blue ones, never undercut them.

Reading: OpenStax Introductory Statistics, Sections 12.1 to 12.3.

12.2 Correlation

The correlation coefficient measures how tightly points hug a straight line,

\[r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}},\]

always between \(-1\) and \(1\). Points above-right or below-left of the mean cross push \(r\) up, while the other two quadrants pull it down. Its square, the coefficient of determination \(r^2\), is the share of the variation in \(y\) that the line explains. Three cautions. The two “strong” presets have the same \(r\) with very different slopes, because \(r\) measures tightness rather than steepness. The curved preset follows a perfect pattern yet gives \(r \approx 0\), because \(r\) sees only straight-line association. And no value of \(r\) says that \(x\) causes \(y\).

Reading: OpenStax Introductory Statistics, Sections 12.2, 12.4.

12.3 The line is an estimate

The line you fit comes from one sample, and another sample would give another line. The grey cloud is a whole population of \((x, y)\) pairs with its own line, drawn dashed in orange. Each draw picks \(n\) of those points, fits them, and leaves the fitted line behind faintly. After a few dozen draws the lines form a band. Its width is the sampling variability of the slope, the same idea as the sampling distribution of a mean in Chapter 7. The spread of the slopes shrinks roughly like \(1/\sqrt{n}\), so raising \(n\) tightens the band around the population line. A slope printed to three decimals is still an estimate with a margin of error.

Reading: OpenStax Introductory Statistics, Section 12.4.

12.4 Prediction and outliers

A fitted line predicts \(\hat{y} = a + bx\) for any \(x\) you like, but the data only vouch for the range they cover. Inside that range the prediction is an interpolation. Outside it is an extrapolation, shaded here as a caution zone, where the relationship may bend, level off, or stop making sense. Slide the prediction point past the last observed \(x\) and the status line changes its tone. The orange point is an outlier waiting to happen. Near the other points it barely matters. Dragged far to the right it gains leverage. One point tilts the whole line and drags \(r\) with it. Compare the solid line with the dashed one fitted without it, and always look for such points before trusting a slope.

Reading: OpenStax Introductory Statistics, Sections 12.5, 12.6.