Why Most A/B Tests Fail and the Pre-Mortem That Fixes It

The Diagnosis: Why Tests Fail Before They Start

The standard A/B testing workflow looks like this: PM surfaces a problem, designer mocks up a solution, engineer builds two variants, data team sets up tracking, test runs for two weeks, team reviews results.

If the treatment wins by 5%, ship it. If not, iterate.

This workflow is not wrong. It is incomplete. It treats the test as the decision point. But the decision was made long before the test started.

The hypothesis—the causal story connecting a change to an outcome—is where experiments live or die. Most teams do not write hypotheses. They write features. A feature called "improved checkout flow" is not a hypothesis. It is a guess dressed in a noun.

When teams skip hypothesis formation, three failure modes appear consistently.

First, tests measure the wrong metric because the team optimized for activity rather than outcome. Second, tests conclude prematurely because the team did not define a stopping rule or sample size. Third, tests produce inconclusive results because the effect size was too small to detect, which is itself a signal that the hypothesis was wrong, not that the test needs more time.

These are not statistical problems. They are diagnostic problems. The test is a symptom, not the cause.

The structural fix is to move the failure point upstream. The pre-mortem process does this by asking one question before any test is built: what could cause this experiment to fail? The answer to that question, documented before the first line of code, changes everything about how the test is designed.

The Pre-Mortem Framework for Experimentation

The pre-mortem is a technique from project management, introduced by psychologist Gary Klein. Before starting a project, the team imagines it has failed. Then they work backward: what caused the failure?

The value is not in the imagination exercise. It is in the forcing function. When you must identify failure modes before you act, you are forced to think about causality rather than just outcomes.

Applied to A/B testing, the pre-mortem has four components. Each one maps to a common failure mode.

1. Cause Identification

Before building a variant, write the causal story. Not "this will improve activation," but "users do not complete activation because they abandon at the plan selection step, likely because the annual/monthly toggle creates decision paralysis, so removing the toggle and defaulting to monthly with an upgrade prompt will reduce abandonment by making the decision smaller."

That is a hypothesis. It has a cause mechanism, a predicted behavior, and a metric. The test is designed to confirm or deny that specific chain, not the general aspiration.

The insight: A hypothesis written as a causal chain tells you exactly what to measure and what a winning result looks like before the test starts.

2. Failure Mode Mapping

Once you have a hypothesis, ask what would cause it to fail even if the test runs correctly.

Common failure modes at mid-market SaaS companies include: the metric you chose is not driven by the change you made (correlation without causation), the effect only appears for a subset of users you did not segment for, the winning variant creates a secondary regression in a downstream metric, or the test runs long enough to detect the effect but your users have already adapted to the change and the effect has dissipated.

Each of these failure modes has a pre-mortem response. Segment your analysis by cohort. Add guardrail metrics for downstream behavior. Set a maximum test duration. The pre-mortem does not eliminate uncertainty. It converts unknown unknowns into known unknowns, which you can design around.

The insight: Mapping failure modes before building the test changes the build itself. Engineers and designers make different choices when they know what the test is actually testing.

3. Sample Sensitivity Analysis

Most teams run power calculations backwards. They ask "how long do we need to run this test to reach significance?" instead of asking "what minimum effect size would make this test worth shipping?"

The second question is the right one. If the minimum detectable effect is 15% improvement on a metric that moves by 2% per month organically, the test is not worth running. Not because you lack traffic. Because the hypothesis is too ambitious for the signal-to-noise ratio you have.

The pre-mortem asks this question at the design stage, not the analysis stage. It saves weeks of running a test that was structurally unlikely to produce a result.

The insight: Sample sensitivity analysis is not about having enough traffic. It is about calibrating your hypothesis to the signal you can actually measure.

4. Decision Threshold Definition

Define the decision threshold before the test starts. This means three things: what metric decides the outcome, what minimum effect size counts as meaningful, and what happens if the result is inconclusive.

The last point is underused. Most teams treat inconclusive results as failures. They are not. An inconclusive result is information: the effect size is smaller than your test was powered to detect, which means either the hypothesis was wrong or the test was underpowered. Both are actionable. Neither requires a winning variant.

The insight: Defining decision thresholds before the test prevents the most common post-test failure: running the test longer because the team wanted a different answer.

What the Data Shows About Experimentation ROI

The framing that most A/B tests fail is common, but the failure is rarely in the execution. It is in the upstream decisions. Teams that invest in hypothesis quality see different outcomes than teams that invest in tooling.

Research across product-led growth companies shows a consistent pattern: the gap between high-performing experimentation programs and low-performing ones is not the number of tests run. It is the quality of the questions being asked. Teams that treat experimentation as a decision-making process, not a testing process, ship fewer changes but move metrics more reliably.

This figure appears consistently across companies that link their experimentation programs to revenue metrics rather than vanity metrics. The pattern is structural: when the hypothesis is tied to a business outcome, the test measures the right thing, and the result is actionable regardless of which variant wins.

The companies that see the highest ROI from experimentation share one characteristic. They have a pre-mortem process. Not a formal document. A structured conversation that happens before any test is built. The conversation forces the team to agree on what success looks like, which is harder than it sounds.

"The most expensive mistake in product development is building something that works but does not matter. Validation before building is not a luxury. It is the only way to avoid that mistake at scale."

— Amplitude Product Team

The table below maps the common failure patterns against their root causes and structural fixes. The pattern is consistent: most failures are upstream of the test itself.

The structural fixes are not complex. They do not require new tools. They require a process that happens before the test is built. That is the pre-mortem.

What to Do Instead

The obvious alternative to the pre-mortem process is more testing. Run more tests. Use a better tool. Hire a data scientist. This approach is not wrong. It is incomplete. More testing without better hypothesis formation produces more inconclusive results, not more progress.

Another common alternative is to abandon A/B testing entirely and rely on qualitative research. User interviews, session recordings, and usability tests are valuable. They are not a substitute for experimentation. The question is not testing versus not testing. It is whether your tests are designed to answer the questions you actually have.

A third alternative is to copy what worked at other companies. A large enterprise changed their signup flow and saw a 20% lift. Your signup flow is different, your users are different, and your context is different. The variant that worked there will not necessarily work here. The pre-mortem process does not prevent copying. It makes the copying more deliberate by forcing you to ask why it worked there and what would need to be true for it to work here.

The right alternative is to treat hypothesis formation as the core skill, not the secondary task. The teams that move fastest are not the ones running the most tests. They are the ones with the clearest picture of what they are trying to learn and why. The pre-mortem is the tool that builds that clarity before the engineering work starts.

Concretely: before your next sprint planning, take 30 minutes on each proposed experiment. Write the causal hypothesis. Map the failure modes. Define the decision threshold. Then build. The overhead is 30 minutes. The reduction in wasted sprint time is measured in weeks.