Correlation and linear regression

Everything else in this branch spoke about one quantity: its mean, its spread, its distribution. This lesson is about two, measured on the same objects — floor area and price, speed and fuel consumption, temperature and ice-cream sales.

There are two questions. Whether the quantities move together — that is what the correlation coefficient answers. How — that is what the regression line answers.

The scatter plot

As usual in this branch, it starts with a drawing. Each object contributes one point $(x_i, y_i)$:

The scatter plot is the first step, not an ornament. Every number below can be computed without looking at the picture, and every one of them can then lie — which is what the section on abuses is about.

Pearson's correlation coefficient

The starting point is the sum of products of deviations from the two means:

A product is positive when both coordinates deviate the same way (both above their mean or both below) and negative when they deviate oppositely. The sum therefore says whether agreement outweighs disagreement — but it depends on the units, so it has to be normalised:

Built this way, the coefficient has three properties worth remembering: it always lies in $[-1; 1]$, it has no units, and it does not change when the data is rescaled (converting a price from pounds to euros does not move $r$ at all).

The regression line

Correlation says "strongly" but does not say "by how much". That is what the regression line answers — one chosen line $\hat{y} = ax + b$, picked so that the sum of the squared vertical distances of the points from it is smallest. Hence the name: the method of least squares.

The second formula says more than "how to compute $b$": rearranged it reads $\bar{y} = a\bar{x} + b$, i.e. the regression line always passes through the point $(\bar{x}, \bar{y})$. That makes a convenient check on the arithmetic.

A warning about reading the slope. "A one-unit rise in $x$ corresponds to a rise in $y$ of $a$ on average" is a correct sentence. "A one-unit rise in $x$ causes a rise in $y$ of $a$" is a statement about causation, and regression does not license it — as we shall see.

Residuals

The points do not lie on the line and are not meant to. The difference between an observation and a prediction is called a residual:

The residuals always sum to zero — that follows directly from the formula for $b$ — so the sum is not the point. The pattern is: residuals scattered without structure say the linear model fits; residuals bending into an arc say the relationship is curved; a fan widening to the right says the spread grows with $x$. Those are faults that no single number will show.

The coefficient of determination

The square of the correlation coefficient has its own, very practical reading:

For our data $r = 0.9$, so $r^2 = 0.81$: floor area, speed or whatever plays the part of $x$ accounts for 81 per cent of the variation in $y$, and 19 per cent stays in the residuals.

It is worth seeing how fast that falls:

A correlation of $0.7$ sounds solid and explains less than half the variation. A correlation of $0.3$, which the table above calls "moderate", explains nine per cent — practically nothing.

Prediction and the limits of extrapolation

Any $x$ can be substituted into a regression line to get a number. Not every such number means anything.

• Interpolation — predicting for an $x$ inside the range of the data. There the model has support in observations and is credible.
• Extrapolation — predicting outside it. The line continues there only because lines always continue, not because the relationship stays linear.

Is the correlation significant

A coefficient computed from a sample is — like every statistic in this branch — a random number. The question "is there any correlation in the population at all" is a hypothesis test from the previous lesson, with $H_0: \rho = 0$:

The statistic follows a Student t distribution with $n - 2$ degrees of freedom. In practice it is handier to read off the critical $|r|$ at $\alpha = 0.05$:

The table says two things at once and both matter. A high correlation from a handful of points means almost nothing — our $r = 0.9$ at $n = 5$ barely clears the boundary of $0.88$. And conversely: a low correlation from a thousand points can be significant and entirely unimportant — significance says "this is not noise", while $r^2 = 0.04$ says the model explains four per cent.

Where this mathematics gets abused

This is the lesson after which most people draw the wrong conclusion, so there are five abuses.

1. Correlation read as causation

The most famous sentence in statistics and still the most often broken. The same correlation between $X$ and $Y$ is explained by four different things:

• $X$ affects $Y$ — what we usually assume;
• $Y$ affects $X$ — reverse causation ("hospitals are dangerous, more people die in them");
• both depend on a third variable;
• the coincidence is chance, especially when many pairs were compared at once.

The correction: the arithmetic cannot tell those four apart, because all it sees is a pair of numbers per object. What can settle it is a randomised experiment or domain knowledge — never the coefficient itself, even at $0.99$.

2. The lurking variable

A special and very common case of the above. Ice-cream sales correlate with drownings because both rise in hot weather. The number of firefighters at a fire correlates with the damage because both depend on the size of the fire. A lurking variable manufactures a correlation that no direct link accounts for.

The correction: before believing in a relationship, list the candidates for a third variable. The formal tool is a partial correlation or a model that controls for it; the informal one is the question "what else changes along with both".

3. Extrapolating beyond the data

A model fitted between $70$ and $130$ knows nothing about $200$. The classic examples are dramatic: linearly extrapolating 100-metre sprint records predicts a negative time, and a child's growth predicts a two-metre five-year-old.

The correction: the range of the data is part of the model and is quoted together with the equation of the line. A high $r^2$ does not license extrapolation — it speaks only to the quality of the fit where the points were.

4. The coefficient read without the plot

$r$ measures linear dependence only. Points lying exactly on a parabola can give $r = 0$, and a single outlier can lift $r$ from zero to $0.8$ or drop it from $0.9$ to zero.

The correction: the scatter plot is compulsory, not optional. Four data sets with identical $\bar{x}$, $\bar{y}$, $r$ and regression line but entirely different shapes are the classic Anscombe quartet — built precisely to make this visible.

5. Significance mistaken for strength

On a large sample any non-zero $r$ becomes significant: at $n = 100$ the boundary is $0.20$, at $n = 10\,000$ it is $0.02$. A headline reading "researchers found a significant association" therefore says nothing about how strong that association is.

The correction: read $r^2$, not the p-value. A significant correlation of $0.05$ explains one four-hundredth of the variation and predicts nothing.

Exercises

The first set works on a five-point table, exactly like the examples above. The r prompt gives five pairs and asks for the correlation coefficient — compute it through the deviations from the means, since $\bar{x}$ always lands on the middle point. The "regression line" prompt gives the same data and asks for both coefficients at once: write them as a = …, b = …; the parts are named, so the order does not matter.

The second set is authored, because it asks about quantities the generator deliberately does not compute: $r^2$ is $r$ squared, and a prediction $\hat{y}$ with $a$ and $b$ known is a substitution into a linear function — both already have generators in other branches. They stay here as questions about interpretation, alongside residuals and the arithmetic of extrapolation.

The last question is a trap and is meant to be one: the arithmetic gives $17.6$ and is correct, while the result is worthless, because $200$ lies outside the range of the fit. Judging "may this be extrapolated" has no number for an answer and stays content — it is the third point in the section on abuses.

Common mistakes

• Inferring a cause from a correlation alone — the same number is explained by four different causal arrangements, and the arithmetic does not tell them apart.
• Forgetting the lurking variable — ice-cream sales and drownings correlate through the heat; look for a third quantity before believing in a link.
• Reading $r$ without a scatter plot — the coefficient measures only linear dependence, and a single outlier can invert it entirely.
• Confusing $r$ with $r^2$ — a correlation of $0.7$ explains $49$ per cent of the variation, not $70$; that is the difference between "a strong link" and "less than half".
• Extrapolating beyond the data — lines always continue, data ends where it ends, and a high $r^2$ does not change that.
• Measuring distances perpendicular to the line rather than vertically — the method minimises deviations in $y$, because $y$ is what is being predicted; the perpendicular gives a different line and answers a different question.
• Treating significance as strength — at $n = 10\,000$ an $r$ of $0.02$ is significant, and it explains four ten-thousandths of the variation.
• Swapping the roles of $x$ and $y$ — the regression of $y$ on $x$ and of $x$ on $y$ are two different lines; they coincide only when $|r| = 1$.