Leave a comment
Get the GH Bookmarklet

Ask GH

  • SS

    Saulo Segurado

    over 3 years ago #

    That you need, at least, 100 conversions per variation to start considering the results of your tests.

    The margin of error of smaller number are too risky to justify putting your money into a hypothesis based of the test results.

    Actually that is very counterintuitive since most of people believe traffic is the key factor for A/B tests.

    • JA

      Jean-Baptiste Alarcon

      over 3 years ago #

      True, smaller number aren't reliable. But when you have lots of traffic, you achieve quite rapidly a good enough amount of conversions say in a day or two.

      Should you stop the test here, you could still be getting imaginary results as you didn't test for long enough to have a representative sample and account for seasonality.

      That's why you see the recommendation of testing for full weeks at a time, and for a duration of 1-2 business cycle.

      What's your take on that?

  • JG

    jordan gutierrez

    over 3 years ago #

    Jquery :D

  • JG

    jordan gutierrez

    over 3 years ago #

    Also, Optimizely could be a bit funky. Take a look at this http://i.imgur.com/C5FNC09.png . There was a "traffic spike" but actully nothing happen. Thankfully we do assign attributes based on the A/B test and we didn't noticed any change. Have a backup plan in case Optimizely goes bad (Woorpa is out backup tool)

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
6
6