Arrow icon
Ness Labs: Make the most of your mind
Learn more about Joggo

A Summary of

What every product manager should know about AB Testing

James Copeman
View original

As a product manager, one should understand the following about A/B testing.

Types of testing

  1. A/B Tests – Directly compares one user experience to another
  2. A/B…N Tests – Comparing multiple variants of user experiences
  3. Multivariate Tests – Testing multiple variables in one test

Analyzing results

  • Traffic is important for measuring the difference in behaviors across users
  • Most changes will result in small changes in outcomes
  • Larger sample sizes filter out random variation

What to measure

  • Frame a hypothesis with an “if,” “then,” and “because”
  • A primary metric should be highlighted by the “then” clause
  • Secondary metrics are important but may not provide the significant answer to whether a variant is better
  • Insignificant or negative results are not bad; they indicate the test was able to prevent the implementation of a negative functionality
Related content
See all posts
Arrow icon