Leave a comment

The more people you have at your company running good experiments, the better. However, not all of us have a data scientist that can look over out shoulders when running tests to mitigate errors. So it can be daunting for someone new to testing. I wanted to create a guide to help democratize A/B testing and to educate ambitious & aspiring experimenters, and I wanted it to serve two purposes: -A quick reference checklist that anyone, new or experienced, can use to reduce errors in experiments -A central and comprehensive guide to educate people in experimentation topics. Hope you enjoy! It's a bit cumbersome at 6k words, but I made it a sort of "choose your own adventure" based on your level and what topics you care about.

  • SM

    Shanelle Mullin

    5 months ago #

    Alex, you never disappoint. Killer resource.

  • RF

    Ryan Farley

    5 months ago #

    This post is awesome - absolutely sharing it with my team. It's amazing how even very large companies make basic A/B errors like 'hitting significance'. Probably due to an overwhelming volume of total crap written on the subject. We definitely need more stuff like this to drown out the garbage.

    • AB

      Alex Birkett

      5 months ago #

      Probably because many practitioners don't write, and many writers don't practice what they write about. But thanks! I appreciate it, as well as your early thoughts on the checklist/guide.

  • ML

    Mark Lindquist

    5 months ago #

    Love the way you organized it by basic, intermediate, and advanced. Makes it useful for someone at any level. Great stuff as always.

    • AB

      Alex Birkett

      5 months ago #

      Thanks! At first it was just going to be a brief checklist, but brevity isn't a strength of mine. Glad to make it at least scannable/searchable in some better way than a wall of text

  • MM

    Marcus Meissner

    5 months ago #

    Finally something good on A/B testing. I am quite fed up with reading case studies about 131% lifts (that obviously never happened) from the infomarketers of the world. The only thing is it is quite long and I found myself jumping around. I'll have to bookmark it and come back to it as a reference.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
26
26