Leave a comment

While 'best practices' exist to help improve performance, they may not always be optimal. Every situation calls for a different solution. This article shows how three 'best practices' were debunked and invalid for a project recently done by the author. The following three are discussed:


  1. Your headline must speak to your visitors’ pain

  2. People won’t provide an email address for something that should be freely available

  3. You need 250+ conversions to declare a champion

  • JC

    Jason Culbertson

    almost 6 years ago #

    I don't understand how the homepage is a fair A/B test. The registration moved down the page. A proper test would have left the design exactly the same and simply changed the headline.

    • JG

      Jen Gordon

      almost 6 years ago #

      hi Jason since we had low traffic, we decided to test bigger, more dramatic changes - another "best practice" of course with low traffic pages, but making the dramatic changes told us a lot more than making one change and leaving the rest as is -- hope this helps - jen :)

  • SS

    scott sharp

    almost 6 years ago #

    I'm really scratching my head on this one...

    "When you don’t have a ton of traffic to send to your A/B tests, big changes have more noticeable and measurable effects. Making dramatic changes per variant helps you get the data you need even if we can’t send more traffic (or wait until 2054!)."

    I'm extremely skeptical that the types of on-page changes shown here affect variance so much that you should put any weight into a 8 conversions vs. 5 conversion outcome test result. The point of A/B testing is to provide statistical evidence that one thing is better than another thing. There is no spinning the underlying statistics. If you don't have enough traffic, you aren't going to get a conclusive A/B test.

    Running page changes as A/B tests in these cases is largely a dog and pony show. In the long-run, you're not going to make any smarter of choices than if you hadn't run the test at all.

    • JG

      Jen Gordon

      almost 6 years ago #

      Hey Scott - Unfortunately there isn't the time or probably reader attention span to write to the depth I would have liked on the findings we had. I had to summarize a lot. At the end of the day the changes resulted in 30% more tangible leads for NueMD sales to follow up on. That's not a dog and pony show. :) Was it absolutely the result of a headline change? We can only keep testing over time to make sure.

      • SS

        scott sharp

        almost 6 years ago #

        Hey Jen, I don't want to sound like I'm diminishing the end results. I think my point was that the A/B test results can't be considered conclusive.

        I'll browse through this site and Inbound.org a few times a week and it seems like there is a steady stream of posts along the lines of "Why your A/B test results don't hold up", "How to avoid A/B test mistakes", etc. And most boil down to improper measurement or inadequate sample breadth and size. That suggests to me that there is inadequate information out there. Couple that with the testing platforms I've used not doing a particularly good job of guiding you towards real, legitimate results, and I think there is a real deficiency of real A/B testing improvements.

        That means there are lots of false positives (I guess this these are the "dog and pony show" results I'm talking about, where you can show your boss that the site improved, but the final numbers don't add up months down the road). So when I see a post like this where it's very easy to come across a false positive, I'm inclined to comment :)

        This isn't to say your results aren't real, just that there is uncertainty. As you said, keep testing over time to make sure. And a win's always a win!

        • SC

          Shana Carp

          almost 6 years ago #

          due the structure of a frequentist test, actually, you can say that the results aren't real.

    • SC

      Shana Carp

      almost 6 years ago #

      this would have been a good place for a bayesian bandit test - just throw all the choices in, let it run.

SHARE
14
14