Leave a comment

There are two main types of A/B tests that our customers conduct: design changes and copy changes. Copy changes test the content or phrasing of text on the site, whether titles, headlines, paragraphs, or even the “Click Here to Learn More” language on a button. Design changes include changes to the color scheme and layout of the page, adding or removing elements like images and buttons, and changes to the functionality of the page. In the article we included our own data to show what types of tests companies are running and how successful they are.

Most Impactful Idea to Test from this Article

  • JC

    Janet Choi

    almost 4 years ago #

    Thanks for sharing this @centenaro! Helpful to cut through the myths and temptations of A/B testing... (especially that less than a quarter of tests are statistically significant). Confirms my gut feeling that so many people just love testing for the sake of testing and good tips on what to do when you *do* test!

    • LC

      Luiz Centenaro

      almost 4 years ago #

      So glad you found this useful Janet. I still encourage testing for the sake of testing rather than just blindly implementing things.

      That way the culture of the company develops into a testing culture.

  • AA

    Anuj Adhiya

    almost 4 years ago #

    Good piece.

    The one paragraph I'm not on board with is
    "...radical changes tend to be more successful. This may be because they are more likely to have a clear hypothesis behind them, while incremental changes are more speculative"

    There's 3 parts that didn't jive with my experience

    a. "radical changes tend to be more successful".
    I think what the author means is that radical changes - when they work - tend to be more IMPACTFUL.
    Whether its a radical change or an incremental one, at the end of the day, a win is a win. It's the degree of impact that's different.

    And of course it makes sense that a radical change would likely have more impact because at that point you're discovering something completely new (that's led to a big win) - it's not an optimization of an existing flow/behavior which more often than not will have a smaller impact than the original test than unearthed this new avenue.

    This bleeds into the next point...

    b. "Incremental changes are more speculative"
    I'd argue that it's more often the other way around.
    Big changes are more likely to be speculative because these are "discovery" type tests where you're trying something very new. You're changing a whole host of variables. These are the kinds of tests that give you a directional signal of high impact. It's only when you've discovered something new that works that you can develop a more refined hypothesis for the incremental "optimization" type tests that follow. By definition, you are being less speculative as you refine hypotheses.

    And on the topic of hypotheses...

    c. "Clear hypothesis": Whether the change is radical or incremental, if there isn't a clear hypothesis then you're not likely to be running a good test/to be able to analyze/measure impact correctly. So - as written - implying that tests that involve radical changes may have clearer hypotheses than incremental ones doesn't feel like a true statement to me.

  • JQ

    Jason Quey

    almost 4 years ago #

    I don't think I've ever seen anyone write this. Thanks for sharing that with us Luiz!

  • EL

    EJ Lawless

    almost 4 years ago #

    We have a platform that does workflow management for A/B testing. The platform makes it easy for us to see what gets tested and what the results are. If there specific questions, let me know and I'll look at the data.

  • NR

    Nawaaz RhMaan

    almost 4 years ago #


    Excellent post on A/B testing. The data and graphical representation is perfect.
    You mentioned "With a 26% success rate, changes to the website copy tend to be the most successful " Is it for a simple website or e-commerce site? Does it vary from site to site ?