Leave a comment
Get the GH Bookmarklet

Ask GH

Common answers I hear from companies are below. I'm curious what prevents others from running more tests. And if you feel like you are running enough tests, please tell us how many you run per week. Thanks!

a) lack of big wins from past A/B tests

b) team prioritizing all dev resources for product road map rather than testing

c) too hard to run meaningful A/B tests

d) something else (please specify)

  • JA

    Justin Adelson

    10 months ago #

    In my experience, A/B Testing wasn't done for the following two reasons:

    1) Not enough budget or traffic to attain enough data to make data-driven optimizations.

    2) Unaware of how amazing and valuable A/B testing can be. - Most of my clients are ambitious and like to run their own marketing campaigns but most of the time they are not creating hypothesizes or trying to learn new information - they are just trying to throw a bunch of spaghetti against the wall to see what sticks. The best example of this was an app client who was not following Facebook character guidelines for mobile ads; by shortening their ad copy within FB's guidelines I reduced their CPC by $0.30 and their $/Install by 20% - this test was run along side their existing copy for an accurate A/B test.

  • RB

    Robert Bausmert

    10 months ago #

    At the moment it's simply the lack of traffic. We are pretty early stage and A/B testing is not something to worry about at this stage.

    That being said, the people I work with don't grasp the concept of high tempo testing yet so I doubt that a lot of A/B testing will ever happen unless I push for it. Once we are at the level to run meaningful A/B tests I will try to find some low hanging fruit to get people on board. We'll see where things go from there.

  • MV

    Maja Voje

    10 months ago #

    The speed of implementation. Most companies do not have a devoted growth teams. There are negotiations for executing every experiment, which takes a lot of energy and ends up in having to deal with people who think that experiments cannibalize their time to do "their real work". But it is all a journey of creating a growth culture :)

  • CB

    Charlie Benkendorf

    10 months ago #

    D) Compliance. I work in a regulated industry, and all marketing communications must be reviewed by legal, compliance and our third party bank. A test that at an unregulated company would take half a day to execute on takes 2-3 weeks here to go through legal checks and reviews.

  • BP

    bhola prasad

    10 months ago #

    Hard to do if you don't have knowledge about HTML/CSS. But there is a tool, people can use with any programming knowledge - Visual Website Optimizer - https://vwo.com/ . [ Not related to the company or affiliate, just love the product]

    • SE

      Sean Ellis

      10 months ago #

      Yep a bunch of tools make it much easier. Optimizely, Google Optimize, etc. It'll be interesting to see what happens with VWO and Optimizely now that Google is giving away something that is comparable for free.

  • BP

    bhola prasad

    10 months ago #

    sorry it's No

  • GC

    Gary-Yau Chan

    10 months ago #

    For marketing a/b testing, its easier. For product a/b testing, its more difficult.

    • GC

      Gary-Yau Chan

      10 months ago #

      From my experience, in a small team, even with dedicated engineers, maximum 2 product features can be tested per week.

      The data aspect can be complicated, things to keep in mind: prevent stacking experiments, MDE / run time / wait till statistical significance, and inconclusive results. Coupled with urgency, patience, small wins, resource constraints, and endurance.

      Love to hear frameworks around this challenge.

  • TD

    Tomasz Duda

    10 months ago #

    Other than what has been mentioned before - the ability measure and/or set up proper analytics so that the test in question is measured properly and is going to provide solid answers.

  • MT

    Muhammed Tüfekyapan

    9 months ago #

    I think the main reason is decision makers do not understand value of testing process so they don't prioritise A/B testing in their day to day operations. All of "problems" can be solved but they do not care enough to make A/B testing their number one prioritise.

    "We start our design project, when it is done, we can start to testing." This is the most ironic answer I got. I'll never forget this.

  • RP

    Regis Poulenc

    9 months ago #

    - Too many stuff going on, running at the same time to have relevant outputs. (results are "polluted")
    - Lack of traffic

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
12
12