At a large company, every change can be an experiment. At startups with little data, the opposite is true.
Why run an experiment?
- Learning: experiments help you learn about your users
- Deciding: learn if the change had the intended consequence
- Avoiding: catch unintended consequences
- Quantifying: the impact of your changes
- Aligning: settle subjective debates
What are the downsides to running an experiment?
- Time: experiments take time to set up & run
- False confidence: can create false confidence based on misinterpreted results
- Short-term thinking: they can push you to think short-term
- Narrow thinking: they disincentivize taking bets that are hard to measure
- Bad product: they can introduce awkward UX or legal risk
When to not run an experiment?
- It'll take too long to get actionable results:
- Is it worth the time until you get conclusive results?
- The downside risk of the change is low & effort is high
- What if you instead invested the time into a better experimentation framework?
- You're launching something completely new
- No control group with a new product!