Explore the Two-sample t-test

Pooled vs Welch Simulator

Try a comparison:

1Two populations

H₀: μ₁ = μ₂. Group 2 has the same shape as group 1, stretched by σ₂/σ₁ and moved by δ.

2Estimated standard error

Each test divides x̄₂ − x̄₁ by an estimate of its SE. Shown as estimate / true SE, so 1 is a perfect estimate.

3Pooled t and Welch t

4Rejection rate vs σ₂/σ₁

Three tests on the same experiments. With δ = 0 every rejection is a type I error, so a correct test sits on α.

How to use this simulator

What you see. Each experiment draws n₁ values from group 1 and n₂ from group 2 (panel 1) and tests H₀: μ₁ = μ₂ from the difference x̄₂ − x̄₁. Its true standard error is SE = √(σ₁²/n₁ + σ₂²/n₂), but the σ's are unknown, so each test estimates SE from the samples (panel 2) and divides by it (panel 3). Panel 4 compares how often the tests reject, for every σ ratio you have run.

The three tests.

  • z, σ known — divides by the true SE. Not possible in practice; shown as the benchmark.
  • Pooled t (Student's) — assumes σ₁ = σ₂ and averages the two variances, weighted by n − 1: sp² = ((n₁−1)s₁² + (n₂−1)s₂²)/(n₁+n₂−2). SE = sp√(1/n₁ + 1/n₂), df = n₁ + n₂ − 2.
  • Welch's t — keeps the variances apart: SE = √(s₁²/n₁ + s₂²/n₂), with the Welch–Satterthwaite df computed from the data (between min(n) − 1 and n₁ + n₂ − 2).

Why the pooled test can fail. When σ₁ ≠ σ₂, the pooled variance is pulled toward the variance of the larger group. If the smaller group has the larger σ, the pooled SE is too small, t is too big, and H₀ is rejected far too often. If the smaller group has the smaller σ, the pooled SE is too big and the test loses power. With equal n the two tests give the same t and both are nearly exact.

Things to try.

  • Equal σ, equal n — both tests sit on α. Move σ₂/σ₁: with n₁ = n₂ the pooled test stays close to α even when the spreads differ.
  • Small group, big σ — n₁ = 5, n₂ = 20, σ₂/σ₁ = 1/2: the pooled test rejects a true H₀ about 19% of the time; Welch stays near 5%. Click All ratios to see the whole curve: on the other side (σ₂/σ₁ = 2) the pooled test almost never rejects.
  • Welch costs little — with a real effect and equal spreads, Welch's power is almost the same as the pooled test's. That is why many textbooks and R's t.test() use Welch by default.
  • Exponential with small n — both t tests assume normal populations; skewness distorts both of them, as on the one-sample page.

Details. Group 2 is group 1 stretched around its mean by σ₂/σ₁ and moved by δσ₁, so the shapes match and only the spread and the location differ. Panel 4 keeps the results for every σ ratio you have run with the current shape, n₁, n₂ and δ; changing any of those clears it. Its dashed line is the z test's theory; the t tests have no simple formula here, so they are shown by simulation only.

Colours. Red ochre is group 1 and grey group 2. Dark blue is the pooled test and purple Welch's test; grey in panel 4 is the z benchmark. Dashed lines are theory.