What you see. Each experiment draws n₁ values from group 1 and n₂ from group 2 (panel 1) and tests H₀: μ₁ = μ₂ from the difference x̄₂ − x̄₁. Its true standard error is SE = √(σ₁²/n₁ + σ₂²/n₂), but the σ's are unknown, so each test estimates SE from the samples (panel 2) and divides by it (panel 3). Panel 4 compares how often the tests reject, for every σ ratio you have run.
The three tests.
- z, σ known — divides by the true SE. Not possible in practice; shown as the benchmark.
- Pooled t (Student's) — assumes σ₁ = σ₂ and averages the two variances, weighted by n − 1: sp² = ((n₁−1)s₁² + (n₂−1)s₂²)/(n₁+n₂−2). SE = sp√(1/n₁ + 1/n₂), df = n₁ + n₂ − 2.
- Welch's t — keeps the variances apart: SE = √(s₁²/n₁ + s₂²/n₂), with the Welch–Satterthwaite df computed from the data (between min(n) − 1 and n₁ + n₂ − 2).
Why the pooled test can fail. When σ₁ ≠ σ₂, the pooled variance is pulled toward the variance of the larger group. If the smaller group has the larger σ, the pooled SE is too small, t is too big, and H₀ is rejected far too often. If the smaller group has the smaller σ, the pooled SE is too big and the test loses power. With equal n the two tests give the same t and both are nearly exact.
Things to try.
- Equal σ, equal n — both tests sit on α. Move σ₂/σ₁: with n₁ = n₂ the pooled test stays close to α even when the spreads differ.
- Small group, big σ — n₁ = 5, n₂ = 20, σ₂/σ₁ = 1/2: the pooled test rejects a true H₀ about 19% of the time; Welch stays near 5%. Click All ratios to see the whole curve: on the other side (σ₂/σ₁ = 2) the pooled test almost never rejects.
- Welch costs little — with a real effect and equal spreads, Welch's power is almost the same as the pooled test's. That is why many textbooks and R's
t.test() use Welch by default.
- Exponential with small n — both t tests assume normal populations; skewness distorts both of them, as on the one-sample page.
Details. Group 2 is group 1 stretched around its mean by σ₂/σ₁ and moved by δσ₁, so the shapes match and only the spread and the location differ. Panel 4 keeps the results for every σ ratio you have run with the current shape, n₁, n₂ and δ; changing any of those clears it. Its dashed line is the z test's theory; the t tests have no simple formula here, so they are shown by simulation only.
Colours. Red ochre is group 1 and grey group 2. Dark blue is the pooled test and purple Welch's test; grey in panel 4 is the z benchmark. Dashed lines are theory.