Leave a comment
I wrote a pretty long (4,000 words) post on getting started with A/B testing. Many articles I've read don't focus much on A/B testing from an organizational perspective, so I tried to really focus on that (how to convince your boss, how to get setup, etc) Here's what's covered:
  • What is A/B testing
  • Why most companies don't run A/B tests
  • Why you should care about A/B testing
  • Organizational Setup
  • Testing best practices
  • Worst practices
  • Deep dives - great articles for those interested in learning more
  • BL

    Brian Lang

    about 5 years ago #

    “Priority #1: Get your first test running…. Too often companies get caught up in small details”

    Thank you for this! It’s so true. Results, good and bad, speak for themselves and do wonders for evangelizing A/B testing within an organization.

    Regarding your example on the case study on Unbounce’s site, where there was a limited number of visitors & conversions – just note that there is a fundamental difference in how Google Content Experiments determined a “winner” in that example (using a multi armed bandit approach), compared to the more traditional (frequentist) approach using statistical significance and power that you describe in your post.

    • SC

      Shana Carp

      about 5 years ago #

      Sort of - the MAB approach does work in a frequentist way as well (eg:UCB-1) so they would use similar tests along the way as a standard apprach

    • LT

      Luke Thomas

      about 5 years ago #


      Thanks a ton for pointing this out - I'll update it.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.