Leave a comment
Get the GH Bookmarklet

Ask GH

Running multiple UX tests runs the risk of making your data less accurate. However, running tests sequentially until statistical significance can be quite slow. How does your team approach this?

  • HQ

    Hila Qu

    over 4 years ago #

    Great question, and I often wonder about the same thing as well. I usually lean towards speed rather than accuracy.

    Meanwhile, I think for small companies with limited traffic, you should be even more thoughtful about prioritization, in term of deciding where to test. In term of what to test, you should always lean towards big bold test, which requires relatively less time.

    • AD

      Ali Dinani

      over 4 years ago #

      Thanks for answering Hila! Our team leans in the same direction - however, recently I've read a lot of articles favoring the contrary. I'm curious about how the rest of the community tackles this issue.

  • EM

    Elise Maile

    over 4 years ago #

    Good question, and if your website has low traffic numbers then the toss up between speed vs accuracy is a very tough call to make. Adjusting the main goal of the test(s) can allow you to run multiple tests across the site. For example, test 1 on homepage main goal is increased interaction with the altered UX element, test 2 on product page's main goal is progression to next step, test 3 on payment page goal is conversion.

    If your website has more traffic, running multiple tests at the same time is easier as you can segment the audience. I currently work for a travel company, these are some of the ways we segment our audiences:
    Device (desktop/tablet/mobile)
    Traffic source (PPC/email/natural search)
    Customer type (families/couples/groups)
    Destination (Asia/North Europe/America/Canaries)
    Date of travel (Summer/Winter/Month)

    This gives you more time to run the test without it clashing with others. Some of these segments are also an example of early personalisation so you can learn how different customers react to messages and build upon those findings.

    For both these size websites, I agree with Hila's comment about prioritization of tests and it's importance to enable you to get the best ROI on your tool.