Leave a comment
I wrote a pretty long (4,000 words) post on getting started with A/B testing. Many articles I've read don't focus much on A/B testing from an organizational perspective, so I tried to really focus on that (how to convince your boss, how to get setup, etc) Here's what's covered:
  • What is A/B testing
  • Why most companies don't run A/B tests
  • Why you should care about A/B testing
  • Organizational Setup
  • Testing best practices
  • Worst practices
  • Deep dives - great articles for those interested in learning more
  • BL

    Brian Lang

    almost 7 years ago #

    “Priority #1: Get your first test running…. Too often companies get caught up in small details”

    Thank you for this! It’s so true. Results, good and bad, speak for themselves and do wonders for evangelizing A/B testing within an organization.

    Regarding your example on the case study on Unbounce’s site, where there was a limited number of visitors & conversions – just note that there is a fundamental difference in how Google Content Experiments determined a “winner” in that example (using a multi armed bandit approach), compared to the more traditional (frequentist) approach using statistical significance and power that you describe in your post.

    • SC

      Shana Carp

      almost 7 years ago #

      Sort of - the MAB approach does work in a frequentist way as well (eg:UCB-1) so they would use similar tests along the way as a standard apprach

    • LT

      Luke Thomas

      almost 7 years ago #


      Thanks a ton for pointing this out - I'll update it.