Hypothesis testing, the p-value and the two kinds of error

The previous lesson answered the question how much — what the mean is and how precisely we know it. This one answers the question whether: whether an observed difference is real, or within the reach of chance.

A new page variant converts at $4.8\%$ instead of $4.0\%$. A filling machine dispenses $499.4$ g instead of $500$. A drug lowered blood pressure by $5$ mmHg. Every time the question is the same: is this an effect, or is it noise.

Two hypotheses

A test begins with a pair of claims about the population:

The null hypothesis $H_0$ says nothing is happening: the mean equals the nominal value, the variants convert alike, the drug does not work. The alternative hypothesis $H_1$ says something is.

The pair is asymmetric, and that is the first thing to understand:

• $H_0$ always carries the equality, because only then can the distribution of the test statistic be computed. Writing $\mu = 500$ fixes one particular distribution; writing $\mu \neq 500$ fixes infinitely many.
• $H_0$ stands until the data overturns it. A test never "proves" $H_0$; at best it rejects it or fails to. This is the same asymmetry as in a courtroom: innocence is assumed until guilt is shown, and an acquittal does not mean innocence was proved.

The alternative can also be one-sided ($H_1: \mu > \mu_0$), when only one direction is of interest in advance — whether a drug lowers pressure, rather than whether it changes it. The direction must then be fixed before the data is seen; we come back to that in the section on abuses.

The test statistic

Since $H_0$ fixes a distribution, we can ask how unusual our sample is relative to that distribution. The measure of unusualness is the test statistic:

The numerator is the sample's departure from the hypothesis; the denominator is the standard error of the previous lesson. The quotient therefore says how many standard errors the sample mean sits from what $H_0$ claims.

Dividing by $\sqrt{n}$ is the heart of it: the same $0.6$ g shortfall is nothing at $n = 4$ and an alarm at $n = 100$. That step is exactly what separates a test statistic from the plain standardisation of a single observation in the normal-distribution lesson.

The rejection region and the significance level

It remains to settle how unusual a sample has to be to overturn $H_0$. The boundary is set by the significance level $\alpha$ — the acceptable probability of a false alarm.

The decision rule is a single comparison:

These are the same values the estimation lesson matched to confidence levels — and that is no coincidence. We come to the link between the two calculations in a moment.

For the machine in the example $|z| = 3 > 1.96$, so we reject the hypothesis that it dispenses $500$ g.

The p-value

Comparing against a boundary gives a yes/no answer but loses the information about how extreme the result was. That is what the p-value carries:

In words: the probability of observing data at least as extreme as ours, assuming $H_0$ is true. The smaller it is, the worse the data fits the null hypothesis.

The decision rule can therefore be written a second, equivalent way: reject $H_0$ when $p < \alpha$. Three numbers from the three-sigma rule are enough to read p-values without tables:

The direction of the conditioning in the definition is everything, and we return to it in the section on abuses: $p = P(\text{data} \mid H_0)$, not $P(H_0 \mid \text{data})$. That is the same difference as between $P(A|B)$ and $P(B|A)$ in the conditional probability lesson.

The two kinds of error

A decision made from a sample can be wrong in two ways — and they are entirely different errors:

• Type I error — a false alarm: announcing an effect that is not there. Its probability is $\alpha$ and we set it ourselves, in advance.
• Type II error — a miss: the effect exists and we do not detect it. Its probability $\beta$ depends on how large the effect is and how large the sample is.
• Power is $1 - \beta$ — the chance of detecting an effect that really exists. The conventional floor of respectability is $0.80$.

The picture shows what the table cannot: both errors are fighting over the same boundary. Moving the mark to the right (tightening $\alpha$) shrinks the shaded tail and at the same time enlarges the part of the right curve to the left of the mark. The only way to shrink both at once is to pull the curves apart or narrow them — a bigger effect or a bigger sample.

A test and a confidence interval

The two calculations of this lesson and the previous one are the same calculation written from two sides:

A two-sided test at level $\alpha$ rejects $H_0: \mu = \mu_0$ exactly when $\mu_0$ lies outside the confidence interval at level $1 - \alpha$.

That is why more and more journals require an interval alongside the p-value: the interval says how large the effect is and how precisely it was measured, the p-value only whether it cleared a threshold.

Where this mathematics gets abused

This is the most abused calculation in science, so there are five abuses.

1. The p-value read as the probability of the hypothesis

"$p = 0.03$, so there is only a 3 per cent chance the null hypothesis is true." False, and it swaps the conditioning: $p$ is $P(\text{data} \mid H_0)$, while that sentence is about $P(H_0 \mid \text{data})$.

The correction: the right wording is "if $H_0$ were true, data as extreme as ours would turn up in 3 per cent of studies". To say anything about the probability of the hypothesis itself you need a prior and Bayes' formula — and the test does not take that road. For hypotheses that are rarely true a priori, $P(H_0 \mid \text{data})$ can sit in the tens of per cent even at $p = 0.03$.

2. "Absence of evidence" read as "evidence of absence"

"$p = 0.31$, so the drug does not work." No: $p = 0.31$ means this data was not enough to reject $H_0$. Perhaps there is no effect — or perhaps the sample was too small to detect it, i.e. the test had low power.

The correction: the conclusion is "we do not reject $H_0$", never "we confirm $H_0$". To actually demonstrate the absence of an effect there is the equivalence test, or a confidence interval that excludes every practically important effect.

3. Statistical significance mistaken for importance

With a large enough sample every difference becomes statistically significant, because the standard error shrinks like $1/\sqrt{n}$. At a million observations a conversion difference of $0.01$ percentage points gives $p < 0.001$ — and means nothing whatsoever to the business.

The correction: significance says "this is not noise", not "this matters". The effect size and its confidence interval are a separate and equally necessary piece of information; the threshold of practical importance is set by the field, not by statistics.

4. p-hacking and multiple testing

Run twenty independent tests at $\alpha = 0.05$ with every null hypothesis true, and on average one comes out significant. Not because something was discovered, but because that is how $\alpha$ was defined. The same arithmetic sits behind adding observations "until it works", dropping outliers after the fact, testing many variables and hunting for the subgroup where it came out.

The correction: hypothesis and analysis plan fixed before the data is collected, and with many tests a correction for multiplicity — the simplest is the Bonferroni rule, dividing $\alpha$ by the number of tests. Preregistering a study is the practical version of the same principle.

5. Choosing the hypothesis after seeing the data

Switching from a two-sided to a one-sided test after seeing which way the data leaned doubles the stated $\alpha$: a result with $p = 0.08$ two-sided becomes "significant" at $p = 0.04$. The same trick, under the name HARKing, consists of phrasing a hypothesis to fit a pattern that was found and presenting it as though it had been stated in advance.

The correction: the direction of the test, and the hypothesis itself, are part of the plan, not of the result. An analysis that found a pattern cannot confirm it with the same data — that takes a fresh sample.

Exercises

The first set walks the three steps of one test. The z prompt gives $\bar{x}$, $\mu_0$, $\sigma$ and $n$ and asks for the test statistic — remember to divide $\sigma$ by $\sqrt{n}$, because without that step you are standardising a single observation rather than a mean.

The "p-value" prompt gives $z$ and asks for the area of the tail, as a percentage, one-sided in some draws and two-sided in others; both read off the three-sigma rule. The third prompt adds a critical value $z^{*}$ and asks for the verdict: the answer is the symbol of the hypothesis that stands, H₀ or H₁. You may type H0 and H1.

The second set is authored and asks about what the generator does not compute: the link between $\alpha$ and the Type I error, between power and the Type II error, the verdict read straight off a p-value and off a confidence interval, and the arithmetic of multiple testing. Where the answer is a hypothesis, write H₀ or H₁.

Naming which kind of error a particular situation risks has no number for an answer and stays content — the table above settles all four cases.

Common mistakes

• Reading p as the probability of the null hypothesis — $p = P(\text{data} \mid H_0)$, not the other way round; swapping the conditioning is the same slip as confusing $P(A|B)$ with $P(B|A)$.
• Concluding "we proved there is no effect" — a test can only reject $H_0$ or fail to; absence of evidence is not evidence of absence.
• Confusing significance with importance — on a large sample every difference becomes significant, including differences of no practical consequence.
• Setting $\alpha$, or the direction of the test, after seeing the data — either one invalidates the stated significance level; switching to a one-sided test after the fact doubles it.
• Dropping the $\sqrt{n}$ from the denominator — without it you are standardising a single observation, and the test is about a sample mean.
• Repeating a test until it succeeds — twenty tests at $\alpha = 0.05$ produce one significant result on average even when there is nothing to find.
• Tightening $\alpha$ without looking at power — a smaller $\alpha$ always raises $\beta$; both fall together only on a larger sample.