Experiment

An experiment is a structured test designed to validate or invalidate a hypothesis about how users will behave, how a product will perform, or what outcomes will result from a change. Unlike intuition-driven development that assumes solutions will work, experimentation treats product decisions as hypotheses that require evidence. It's a disciplined approach to reducing uncertainty before committing significant resources.

Why it matters

Most product ideas fail. Research consistently shows that the majority of new features don't move the metrics they're intended to improve, and many actually make things worse. Building full features based on untested assumptions wastes engineering time, creates complexity, and can actively harm the user experience.

Experimentation inverts this pattern by testing assumptions cheaply before investing heavily. Instead of building first and hoping it works, teams form hypotheses, design minimal tests, gather evidence, and then decide whether to proceed. This approach dramatically reduces waste and increases the odds that what does get built actually delivers value.

For product teams, experimentation creates a culture of learning rather than assertion. Debates about what will work become tests that provide answers. Strong opinions become hypotheses to be validated.

Anatomy of an experiment

The hypothesis

Every experiment starts with a hypothesis-a testable prediction about what will happen. Good hypotheses are specific, measurable, and falsifiable.

Weak hypothesis: "Users will like the new onboarding flow."

Strong hypothesis: "Changing the onboarding flow from 5 steps to 3 will increase completion rate from 65% to 75%."

The strong hypothesis specifies what change is being made, what metric will move, and by how much. This precision makes it possible to design a test that can actually confirm or reject the hypothesis.

Hypotheses should also articulate why the expected effect will occur. "Shorter onboarding will increase completion because each additional step loses users" provides reasoning that can be examined even if the specific prediction is wrong.

Success criteria

Before running an experiment, define what success looks like. What metric needs to move? By how much? Over what timeframe? With what statistical confidence?

Setting criteria in advance prevents moving the goalposts after seeing results. It's tempting to reinterpret disappointing results favorably, but this undermines the entire point of experimentation.

Success criteria should also include guardrail metrics-things that shouldn't get worse even if the primary metric improves. Increasing conversion by making checkout faster is good; increasing conversion while increasing refund rates might indicate a problem.

The test design

How will the hypothesis be tested? Common approaches include:

A/B testing shows different versions to different users and compares outcomes. This is the gold standard for measuring the impact of product changes but requires sufficient traffic and time to achieve statistical significance.

Painted door tests measure interest in a feature by presenting it as available before actually building it. Click rates on "coming soon" features indicate demand without development investment.

Concierge tests deliver the promised value manually before automating it. If the value proposition is sound, users will engage even when delivery is inefficient.

Wizard of Oz tests present an automated interface while humans perform the work behind the scenes. This tests user interaction without building actual functionality.

Prototype tests put mockups, wireframes, or limited implementations in front of users to gather feedback before full development.

Fake door tests present features to gauge interest, then explain the feature is coming soon and ask for email signup to measure genuine interest.

The test design should minimize effort while providing sufficient evidence to make a decision. Over-engineered tests waste resources; under-powered tests fail to answer the question.

Sample size and duration

Statistical significance requires adequate sample size. Running an A/B test for one day with 100 users won't produce reliable results for most metrics. Determine required sample size before starting, based on the expected effect size and desired confidence level.

Duration also matters beyond simple sample size. User behavior varies by day of week, time of month, and seasonality. Tests should run long enough to capture these variations-typically at least one full week, often two or more.

Analysis and decision

When the experiment concludes, analyze results against the pre-defined success criteria. Did the hypothesis prove true? Did guardrail metrics hold? What unexpected effects appeared?

The decision that follows should be guided by the evidence but doesn't have to be mechanical. A result that narrowly misses statistical significance might still inform direction. A result that hits the success metric but reveals concerning qualitative feedback might warrant caution.

Document the learnings regardless of outcome. Experiments that "fail" by disproving hypotheses are still valuable-they prevent building things that wouldn't have worked.

Types of experiments

Feature experiments

Testing whether new functionality improves target metrics. These range from small UI changes to entirely new capabilities.

Pricing experiments

Testing how pricing changes affect conversion, revenue, and customer mix. Pricing experiments require extra care around customer fairness and communication.

Messaging experiments

Testing how different copy, positioning, or value propositions resonate with users. Often quick to implement and highly informative.

Growth experiments

Testing acquisition, activation, retention, or referral mechanisms. These often involve marketing and product working together.

Operational experiments

Testing changes to internal processes that affect customer experience-support response times, delivery speeds, communication frequency.

Building an experimentation culture

Start small

Teams new to experimentation should start with simple A/B tests on low-risk changes. Build the muscle before tackling complex experiments with high stakes.

Make it easy

Experimentation flourishes when the infrastructure supports it. Feature flag systems, analytics that track experiment variants, and templates for hypothesis documentation all reduce friction.

Share learnings

Experiments benefit the whole organization, not just the team that ran them. Create systems for sharing what was learned, including failed experiments that prevented mistakes.

Tolerate failure

Most experiments fail to confirm their hypotheses. This is expected and healthy-it's how learning happens. Cultures that punish failed experiments get fewer experiments and more false certainty.

Avoid vanity experiments

Some teams run experiments to justify decisions already made rather than to genuinely learn. This theater undermines trust and wastes resources. Experiments should be designed to potentially change decisions, not confirm them.

Common pitfalls

Peeking at results and stopping experiments early when results look good (or bad) biases outcomes. Determine duration in advance and stick to it.

Testing too many things at once makes it impossible to know what caused any observed effect. Isolate variables to generate clear learnings.

Ignoring qualitative signals reduces experimentation to number-watching. Quantitative results tell you what happened; qualitative research helps you understand why.

Over-optimizing narrow metrics at the expense of broader user experience creates local maxima that hurt overall outcomes. Keep the bigger picture in view.

Not running enough experiments limits learning velocity. The best product teams run many experiments simultaneously, constantly generating evidence to inform decisions.

Experiments and product strategy

Experimentation supports strategy but doesn't replace it. Strategic vision identifies directions worth pursuing; experiments test specific implementations of that vision. A team that only experiments without strategy will optimize locally but miss larger opportunities. A team with only strategy and no experimentation will pursue assumptions that may be wrong.

Klero supports experimentation culture by connecting customer feedback to experiment results. When an experiment produces unexpected outcomes, feedback often explains why. When experiments succeed, feedback helps understand what users actually valued. This qualitative layer enriches quantitative experiment data.