Arrow icon
Ness Labs: Make the most of your mind
Learn more about Joggo

A Summary of

When NOT to run an experiment

by
Lenny Rachitsky
Lenny's Newsletter
View original

At a large company, every change can be an experiment. At startups with little data, the opposite is true.

Why run an experiment?

  1. Learning: experiments help you learn about your users
  2. Deciding: learn if the change had the intended consequence
  3. Avoiding: catch unintended consequences
  4. Quantifying: the impact of your changes
  5. Aligning: settle subjective debates

What are the downsides to running an experiment?

  1. Time: experiments take time to set up & run
  2. False confidence: can create false confidence based on misinterpreted results
  3. Short-term thinking: they can push you to think short-term
  4. Narrow thinking: they disincentivize taking bets that are hard to measure
  5. Bad product: they can introduce awkward UX or legal risk

When to not run an experiment?

  1. It'll take too long to get actionable results:
    1. Is it worth the time until you get conclusive results?
  2. The downside risk of the change is low & effort is high
    1. What if you instead invested the time into a better experimentation framework?
  3. You're launching something completely new
    1. No control group with a new product!
Related content
See all posts
Arrow icon
April 11, 2020
/
Ravi Mehta
/
Ravi Mehta

Becoming A Peak Product Manager

Read more
September 1, 2020
/
Marty Cagan
/
Silicon Valley Product Group

Discovery – Learning vs. Insights

Read more
July 11, 2011
/
Andrew Chen
/
Andrew Chen

Don’t compete on features

Read more
/
/
First Round Review

How to Engage Your Users to Build the Product They Actually Want

Read more
August 19, 2018
/
Brandon Chu
/
Black Box Of Product Management

Product Management Mental Models for Everyone

Read more