Explore the p-value

Hypothesis Testing Simulator

Try a comparison:

1Population & hypotheses

H₀: μ = μ₀. Move δ away from 0 to make H₀ false. Drag on the chart to draw your own population.

2Test statistic z

3Distribution of p-values

Each experiment gives one p-value. Bars in colour are p ≤ α: H₀ rejected.

4Errors & power

How to use this simulator

What you see. Each experiment draws a sample of N values from the population in panel 1 and tests H₀: μ = μ₀ with a z test (σ known): z = (x̄ − μ₀)/(σ/√N). Panel 2 collects the z values, panel 3 the p-values, and panel 4 counts how often H₀ was rejected. The slider δ sets how far the true mean μ is from μ₀, in units of σ; at δ = 0 H₀ is true.

The p-value. If H₀ is true, z follows the standard normal curve (dashed in panel 2). The p-value is the area under that curve at least as far out as the z you got — shaded for the latest experiment. A small p says "if H₀ were true, a result this extreme would be rare". It is not the chance that H₀ is true.

Things to try.

  • H₀ true: p is uniform — with δ = 0 every p-value between 0 and 1 is equally likely, so p ≤ 0.05 still happens in 5% of experiments. Those are type I errors: false alarms at rate α.
  • One- vs two-tailed — the same z values, judged two ways. Switching Tails keeps the experiments and only changes the decisions. A right-tailed test catches a positive effect more often, but never one in the other direction: drag δ below 0.
  • Errors & power — with a real effect, a miss (not rejecting a false H₀) is a type II error, rate β; power is 1 − β. Larger N or a larger effect raises the power. Compare the curves for N = 10 and N = 100.
  • A tiny effect, a tiny p — set δ = 0.1 and N = 1000: about 9 tests in 10 reject H₀ with a small p, even though the effect is a tenth of σ. A small p does not mean a large effect.
  • The z test needs the Central Limit Theorem — choose Exponential, N = 5, δ = 0 and a right-tailed test, then run 10,000. The p-values are no longer uniform and the type I error rate is not α, because x̄ is not yet normal. Raise N and it comes right.
  • Fair coin or Fair die — the data are discrete, so x̄, z and p can only take certain values and the p histogram is lumpy.

Details. The shapes on 0–32 are treated as continuous: each bar is a bin of width 1 and a draw lands anywhere inside it, so σ is slightly larger than the bar heights alone suggest. The coin, the die and Bernoulli stay discrete. The theory curves assume x̄ is normal, so z ~ N(λ, 1) with λ = δ√N. Panel 4 keeps the results for every δ you have tried with the current population and N.

Colours. Red ochre is the population and grey the latest sample. In panels 2–4 the colour marks rejections — dark blue for z, dark green for p, purple for the rejection rate — and grey bars are experiments where H₀ was kept. Dashed lines are theory.