Point and interval estimation

The previous lesson ended on the central limit theorem and on a formula whose denominator was $\sigma/\sqrt{n}$. This lesson is about what that denominator is for: reasoning about a whole population from a sample.

The question is a practical one. A thousand people were surveyed and the conclusion is meant to be about thirty million. What exactly may be said — and how precisely?

Sample, population, estimator

The distinction between a population and a sample was introduced by the lesson on variance; here it is a tool rather than a topic. What is worth fixing is the notation, because from here on the two worlds stay strictly apart:

Greek letters are parameters — fixed numbers we do not know. Latin letters are statistics — numbers computed from the sample, and therefore random: a second sample gives different ones.

An estimator is a recipe for estimating a parameter from a statistic. The natural estimator of the population mean is the sample mean:

The hat means "an estimate of". This is point estimation: one number. And there is a fundamental problem with it — for a continuous variable the probability that we hit exactly is zero. The number alone therefore says nothing about how far off we are.

Unbiasedness, or what the n − 1 divisor is for

An estimator is unbiased when its expected value equals the parameter being estimated:

This does not mean it hits — it means it errs equally in both directions, so averaged over many samples it hits. A biased estimator errs systematically one way, and no number of repetitions repairs that.

The sample mean is unbiased, since $E(\bar{x}) = \mu$. The more interesting case is the variance — and here the divisor returns that the variance lesson stated without a full justification:

The reason is this. The deviations are measured from $\bar{x}$, not from the unknown $\mu$. But $\bar{x}$ is the mean of exactly those points, so it lies closer to them than the true $\mu$ does — the sum of squared deviations from a sample's own mean is the smallest such sum there is. Dividing by $n$ would therefore give a number that is systematically too small. The $n-1$ correction is chosen precisely so that

This is called the number of degrees of freedom: given $\bar{x}$, the last deviation follows from the others, because the deviations sum to zero. There are $n-1$ independent pieces of information about the spread, not $n$.

The standard error of the mean

Since $\bar{x}$ is a random number, it has a distribution of its own — the sampling distribution of the mean. The central limit theorem of the previous lesson tells us everything we need about it: it is approximately normal, has the same mean $\mu$, and a standard deviation of

This quantity is called the standard error of the mean, and it has to be kept apart from the standard deviation:

• $\sigma$ measures the spread of the data and does not depend on how much of it was collected;
• $\mathrm{SE}$ measures the spread of the mean and shrinks with sample size.

"People are $175 \pm 7$ cm tall" is a statement about the deviation. "The mean height is $175 \pm 0.7$ cm" is a statement about the standard error. Those are two entirely different claims, and confusing them is the commonest mistake in all of statistical inference.

The confidence interval for a mean

Since $\bar{x}$ is approximately normal with standard deviation $\mathrm{SE}$, the three-sigma rule tells us how far from $\mu$ it usually falls. Turning that sentence around gives the confidence interval:

The coefficient $z$ is chosen to match the required confidence level:

The value $1.96$ is worth recognising: it is the exact version of the "two sigmas" of the 68–95–99.7 rule. Half the width of the interval is called the margin of error:

What "95 per cent" actually means

This is where it is easiest to say something untrue, so it is worth settling with a picture.

The picture shows what a sentence cannot: the interval is random, the parameter is not. The true mean $\mu$ is a fixed number and either lies inside a given interval or does not — there is no probability involved. What is random is the sample, and therefore the interval built from it.

The correct sentence is: the procedure that produced this interval hits in 95 cases out of 100. We return to the incorrect one in the section on abuses, because it is the single most common misstatement in applied statistics.

How many observations are needed

The margin-of-error formula also reads backwards. If we know in advance how narrow an interval will do, then

The square is the crux: accuracy enters the sample size in the second power. An interval half as wide costs four times the sample; one ten times as narrow costs a hundred times.

That is also the answer to why polls are run on samples of about a thousand. The first three percentage points of accuracy are cheap; every one after that costs four times the last.

Where this mathematics gets abused

Four abuses, each with a name, because a named one is easier to spot.

1. "There is a 95 per cent probability the parameter lies in the interval"

The commonest misstatement in applied statistics, and one that appears even in textbooks. It is false in the language we are working in: $\mu$ is not a random variable, so the statement "$\mu \in (31\%;\ 37\%)$" has no probability — it is simply true or false.

The correction: the percentage refers to the procedure, not to a particular interval. Correctly: "if the study were repeated many times, about 95 per cent of intervals built this way would cover $\mu$." The reading in which the parameter is random is a separate approach — the Bayesian one — and there it is called a credible interval, not a confidence interval, and it is computed quite differently.

2. A margin of error with no confidence level

"The margin of error is 3 points" is not complete information: on the same sample, the margin at $99\%$ is half again as large as at $90\%$. Giving the number alone lets the confidence level be chosen after looking at the result.

The correction: a margin of error is always quoted with its confidence level, and the level is fixed before the data is collected.

3. A confidence interval mistaken for a range of observations

"The confidence interval for mean height is $(174.3;\ 175.7)$ cm" does not mean that 95 per cent of people are that tall. It speaks only about the mean, not about any individual — which is precisely why it is so narrow. The range holding 95 per cent of the observations is a different thing entirely, computed from $\sigma$ rather than $\sigma/\sqrt{n}$, and at $n = 400$ it is twenty times wider.

The correction: a confidence interval is about the parameter, a prediction interval about a future observation. Confusing them is exactly confusing $\sigma$ with $\sigma/\sqrt{n}$.

4. A margin of error read as a guarantee of sample quality

This one is the most expensive. The formula $z\sigma/\sqrt{n}$ measures only sampling error — it assumes the sample is random and that everyone drawn answered. It measures neither selection bias, nor non-response, nor a badly worded question. An online survey with fifty thousand responses has a negligible margin of error and may be worthless, because the people who filled it in are the people who wanted to.

The correction: the margin of error is a lower bound on uncertainty, not an upper one. Collecting a larger sample narrows it and removes not one systematic error.

Exercises

The first set follows the order in which such a problem is actually worked. The s/√n prompt gives a deviation and a sample size and asks for the standard error — the one step that separates the spread of the data from the spread of the mean. The "confidence interval" prompt gives $\bar{x}$, $\sigma$, $n$ and $z$, and the answer is a pair of numbers in brackets, left end first: an interval is ordered, so writing it the other way round does not pass.

The third prompt reads the same relation both ways: sometimes it asks for the margin of error at a given $n$, sometimes for the required sample size at a given margin. The sample size comes out a whole number in these questions, because the data is built backwards from the answer.

The second set is authored, and it asks not about the arithmetic but about the behaviour of the interval: how it responds to a change of sample size, how many intervals miss at a given confidence level, how large the bias of an unbiased estimator is. The generator does not draw these, because they are properties of the procedure rather than the result of one problem.

The judgement "may such an interval be computed here at all" has no number for an answer and stays content — the four situations in which the answer is no are in the section on abuses.

Common mistakes

• Confusing $\sigma$ with $\sigma/\sqrt{n}$ — the first measures the spread of the data, the second the spread of the mean; the slip inflates the interval by a factor of $\sqrt{n}$ and is mistake number one in this lesson.
• "There is a 95 per cent probability the mean lies in this interval" — the interval is random, the parameter is not; the percentage describes a procedure repeated many times.
• Multiplying by $z$ before dividing by $\sqrt{n}$ — the order does not change the answer, but omitting the division does; check that the denominator really carries a square root.
• Rounding a required sample size down — $61.5$ observations means $62$; rounding down gives a margin larger than the one asked for.
• Treating a confidence interval as a range of observations — the first is about the mean and is very narrow at large $n$; the range holding 95 per cent of the units is computed from $\sigma$.
• Reading a small margin of error as proof of a sound study — the margin measures sampling error alone and sees neither selection bias nor non-response.
• Using $z$ on a small sample with unknown $\sigma$ — below roughly thirty observations the right distribution is Student's t with $n-1$ degrees of freedom, whose critical values are larger.