No results found for your search
How many % of your experiments ended with "inconclusive"?
Across the board, 20-40% of experiments come out as inconclusive. The biggest reasons are poorly formed hypothesis, too small sample size and unclear success metric. Hence, when an experiment isn't conclusive, the most important step is to determine why, document that and not repeat the mistake.
Thanks for sharing this. If 20-40% of experiments are inconclusive, with % fail, so there is a very small chance of % experiments are success.
How do you reassure the team as well as stakeholders above that this is normal? Or is it normal, how we can improve this?
Well, I think we define to define "fail" first. I think an experiment fails when it's inconclusive. The outcome is not a measure of success or failure of an experiment, only for a metric. So, an experiment can only fail when it doesn't lead to a clear outcome (=inconclusive).
Whether the experiment supports the hypothesis or not is a different question.
I see a trend for most experiments becoming more successful in positively impacting KPIs and metrics because your understanding of your target audience gets better. However, it's there will always be surprises and that's why experiments must always be done.
It's a tricky one. I'd say about 30%, in absolute terms, come out as 'inconclusive'.
However, I tend to take something out of every experiment I run. This 'something' allows me to dig deeper and find gems. I'd say less than 5% of my experiments end up as a total waste of time -- if that.
A good example is this recent experiment I ran. I wanted to test republishing articles on Medium, see if that would bring me consistent traffic. I posited a certain amount of traffic (based on past, limited experience) as my goal. I didn't reach that goal, but learned three super valuable lessons. I wrote a full report of the experiment here, if you are interested: https://medium.com/from-zero-to-grow/analysis-ztg001-grow-traffic-from-medium-63c2fc0d0b92
As we speak, I am running a follow-up experiment on this one which is completely crushing all expectations in terms of results. So, was the first one 'inconclusive'? Probably not.
Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter
Use the feedback box below if you have a question, comment or general feedback.
Your feedback has been sent.
Sweet! The link has been copied to your clip boardy board!
Flash isn't supported. Please copy the link manually.