Leave a comment
Get the GH Bookmarklet

Ask GH

  • KI

    Kevin Indig

    almost 2 years ago #

    Across the board, 20-40% of experiments come out as inconclusive. The biggest reasons are poorly formed hypothesis, too small sample size and unclear success metric. Hence, when an experiment isn't conclusive, the most important step is to determine why, document that and not repeat the mistake.

    • GC

      Gary-Yau Chan

      almost 2 years ago #

      Thanks for sharing this. If 20-40% of experiments are inconclusive, with % fail, so there is a very small chance of % experiments are success.

      How do you reassure the team as well as stakeholders above that this is normal? Or is it normal, how we can improve this?

      • KI

        Kevin Indig

        almost 2 years ago #

        Well, I think we define to define "fail" first. I think an experiment fails when it's inconclusive. The outcome is not a measure of success or failure of an experiment, only for a metric. So, an experiment can only fail when it doesn't lead to a clear outcome (=inconclusive).

        Whether the experiment supports the hypothesis or not is a different question.

        I see a trend for most experiments becoming more successful in positively impacting KPIs and metrics because your understanding of your target audience gets better. However, it's there will always be surprises and that's why experiments must always be done.

  • AD

    Alex D

    almost 2 years ago #

    It's a tricky one. I'd say about 30%, in absolute terms, come out as 'inconclusive'.

    However, I tend to take something out of every experiment I run. This 'something' allows me to dig deeper and find gems. I'd say less than 5% of my experiments end up as a total waste of time -- if that.

    A good example is this recent experiment I ran. I wanted to test republishing articles on Medium, see if that would bring me consistent traffic. I posited a certain amount of traffic (based on past, limited experience) as my goal. I didn't reach that goal, but learned three super valuable lessons. I wrote a full report of the experiment here, if you are interested: https://medium.com/from-zero-to-grow/analysis-ztg001-grow-traffic-from-medium-63c2fc0d0b92

    As we speak, I am running a follow-up experiment on this one which is completely crushing all expectations in terms of results. So, was the first one 'inconclusive'? Probably not.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
10
10