Multiple comparisons

Multiple comparisons arise when you test several hypotheses, variants, or metrics at once. With a fixed alpha threshold, the probability of getting at least one false positive increases with their number. It is controlled with a correction, such as Bonferroni's, or by limiting the tests to those decided in advance.

Example

With 20 tests at alpha = 0.05, you expect about one false positive even with no real effect at all.

Related concepts

The whole glossary