Statistical Trap
The "Peeking" False Positive Rate
Standard p < 0.05
Running the test for the full duration. 1 in 20 chance of declaring noise a winner.
Peeking (5 Stops)
Checking early and stopping when significant. Balloons your chance of shipping noise.
productquant.dev