Leave a comment

The article examine the dangers of ending a test early, it gives reasons why you shouldn't end tests even though your tools may be declaring one variation a winner.

It also gives some guidelines on how long you should run your tests for best results.

  • PL

    Peep Laja

    almost 6 years ago #

    Thanks for sharing!

    How often does it happen to you that when you roll out a winning test variation, you actually don't see a lift on your bank account?

    This post explains why.

  • GB

    Gideon Baldridge

    almost 6 years ago #

    I'm always surprised this doesn't come up more. All the a/b testing suites talk like they have some magical statistical transform that solves this problem. I've never heard a statistician say there's any way to handle this other than choosing a somewhat-arbitrary-and-usually-long window of time before running the test.

  • SS

    scott sharp

    almost 6 years ago #

    I see this as partially the fault of the tester and partially the fault of the testings tools.

    Ideally, before starting an A/B test, you have some historical record of the conversions of your control version. For example, on a page I routinely test, I know I average around 42-45%, so when I run an A/B test, achieve significance on the variant, at 44% and see the control is at, say, 40%, I know that the test needs more time. Until I'm comfortable with the conversion rate of the control, I never trust the results of the A/B test.

    However, A/B service like VWO should not EXPECT me to do that or expect me to be capable of doing that. VWO is already firing on all my pages and from my history, knows what my conversion URLs are. I don't see why they are collected data even when tests aren't running, to gain a historical picture of how a URL performed. This wouldn't be an exact science since changes do get elevated (although, they could account for this)

    • SS

      scott sharp

      almost 6 years ago #

      Hit post too soon!

      I wanted to also add that anyone who is new to A/B tests, run concurrent tests, one as an A/B and one as an A/A,or at least tinker with both types of tests so you become familiar with how your site operates, how A/B testing operates, and how long you should run your tests for to see REAL results.

SHARE
29
29