What you see. Each experiment draws a sample of N values from the population in panel 1 and tests H₀: μ = μ₀. In real life σ is unknown, so it is estimated by the sample SD s (panel 2) and the test statistic becomes t = (x̄ − μ₀)/(s/√N) (panel 3). Panel 4 compares three decision rules on the same experiments, for every N you have run.
The three rules.
- z, σ known — the z test of Explore the p-value: uses the true σ. Not possible in practice; shown as the benchmark.
- t with z cutoff — plugs in s but still rejects at ±1.96 (for α = 0.05, two-sided). This is the mistake the t test fixes.
- t test — rejects at the cutoff from Student's t distribution with N − 1 degrees of freedom, e.g. ±2.78 for N = 5.
Why t. s is itself random. For small N it is often well below σ (panel 2), and then t is too big. So t has heavier tails than the normal curve, and ±1.96 rejects too often. Student's t distribution has exactly the right tails when the population is normal; as N grows, s settles near σ and t(N − 1) becomes the normal curve.
Things to try.
- Why not z? — N = 5: s < σ in about 59% of samples, and the z cutoff rejects about 12% of the time instead of 5%.
- t → normal as N grows — click All N × 1,000: the z-cutoff rule falls toward 5% as N grows, while the t test stays on α at every N.
- Skewed population — the t test assumes a normal population. With Exponential and N = 5 its two-sided type I error is about 12%, the left-tailed one about 16% and the right-tailed one about 1%; even at N = 100 it is not quite 5%. In a skewed population x̄ and s move together (a large x̄ usually comes with a large s), so t is skewed too. Here the z test with σ known does better.
- Move δ away from 0 to compare power: with small N the t test has a little less power than the z test, the price of not knowing σ.
- Fair coin with small N sometimes gives s = 0 (all heads or all tails), so t cannot be computed; the readout counts those.
Colours. Red ochre is the population and grey the latest sample; dark green is s, dark blue the t test's rejections and purple the rejection rates. In panel 3, red ochre bars were rejected only by the z cutoff. Dashed lines are theory, which assumes a normal population.