Leave a comment
Get the GH Bookmarklet

Ask GH

I'm completely on board with Sean's "achieving product/market fit requires at least 40% of users saying they would be “very disappointed” without your product".

However, 40% of how many users?

I understand that there may not be an exact formula given that every product/service is different.
I'm just trying to get a sense of whether we are talking about 40% of users in the 10s, 100s or 1000s here.

  • SE

    Sean Ellis

    over 5 years ago #

    When I've run the survey I look for a minimum of 30 responses and ideally like to see north of 100 responses. 30 responses is when I tend to see the numbers stabilize. Someone actually has told me there is some science behind the number 30 but I just arrived there through observation.

    Technically sufficient sample size is a function of the margin of error you are willing to accept. Here's a calculator if you want to be technically accurate about it http://www.raosoft.com/samplesize.html, but for me the numbers I mentioned above have been fine.

    • SE

      Sean Ellis

      over 5 years ago #

      Just had another thought on this... If you need a little more context on Anuj's question, you can get it from this blog post: http://www.startup-marketing.com/using-survey-io/ .

      In the blog post I suggest that you should use this question as part of your decision to start preparing for growth or not.

      So, if you only have a few hundred users and can't get more than 30 responses, what should you do? A couple of the replies below said wait until you have at least 100 responses. I think it's important to recognize that "inaction" is just as much of a decision just as "action" is. So if you only have 30 responses and 40% of those people say they would be very disappointed without your product, I would recommend that you start on a path to growth. But don't stop collecting responses.

      Fortunately the next step isn't to stomp on the gas peddle and start driving growth. As this blog post explains http://www.startup-marketing.com/the-startup-pyramid/ your next step is to start to optimize conversion rates, fine tune positioning, fine tune your business model, etc.

      Growth doesn't stop while you do those things. So during this time you should be able to start approaching the needed 100 responses to scale confidently. If you don't yet have them, scale tentatively until you do have them. If the feedback at 100 is that only 20% of your users consider the product a "must have" then you can stop trying to scale and go back to customer development driven product iterations.

      Of course this is all just my opinion... Open to feedback and discussion.

      • DA

        David Adeyalo

        over 5 years ago #

        Sean, I feel like the survey.io tool is on point. I've used it before and even blogged about it - ha. Ultimately, so many of us read the Lean Startup, read Lean UX, but we get into a pressure situation. Investors want to see graphs that go up and to the right, and so as a founder/CEO there's a ton of pressure to "put the gas on" not because you feel you have product/market fit, but because you need to raise capital. Or you may have product/market fit in one geographic area or market, but not in another. Let's say you have a traffic app, in NYC it may work well, but then in another city you think the strategy is going to work the same and you "pour the gas on," and then it flops.

        Ultimately, these are the kinds of lessons that will entrepreneur's and founder's startup. I hope these lessons are shared beyond the growth hacking community.

        • SE

          Sean Ellis

          over 5 years ago #

          Thanks for feedback David. I think the urgency to achieve milestones to raise money is a relatively good thing. It keeps it all from being too academic. But it's a fine line between urgency and panic. Panic kills a startup faster than just about anything.

      • AA

        Anuj Adhiya

        over 5 years ago #

        Thanks for that added perspective Sean.
        Today's been a heck of a day in GH school thanks to all who took the time to chime in.

    • MM

      Michael Mullany

      over 5 years ago #

      From a hazy recollection of college statistics, 35 samples is the minimum for a 95% confidence using the t-distribution (which is used for small sample sizes and unknown std deviation) assuming that you're trying to the value of a single variable from that sample. (aka the % of yes's to the question "Would you be very disappointed without this product".)

    • LN

      Liam Nolan

      over 4 years ago #

      Exactly what I've been looking for today - knew GH would have the answer! Thanks

  • CC

    Chris Conrey

    over 5 years ago #

    40% of 10 people is 4.
    40% of 100 people is 40
    40% of 1000 people is 400

    Which would you be more willing to go forward with knowing they'll pay for your product?

    Traditionally I've always suggested not making any decisions on less than 90% margins until you've got close to 100 samples for business level decisions.

    If you want the official math, this blog post my business partner wrote about A/B testing has the hard math equation including standard deviations etc: http://vuurr.com/split-testing-determine-sample-size/

    • NW

      Nick Warner

      over 5 years ago #

      "Traditionally I’ve always suggested not making any decisions on less than 90% margins until you’ve got close to 100 samples for business level decisions." - Agreed

    • SC

      Shana Carp

      over 5 years ago #

      Among the many reasons why I like the bayesian approach - you figure out your accepted delta and you earn responses. Takes back to the "get responses"

  • AA

    Anuj Adhiya

    over 5 years ago #

    Thank you Sean & Chris - your responses were very helpful.

  • DA

    David Adeyalo

    over 5 years ago #

    That's a great question. I wonder if Sean could help us answer that. The answer that you provided in the question details is here: http://www.startup-marketing.com/using-survey-io/

    In the past, I've seen people do customer development surveys with as few as 5 people, and as many as several thousand, but typically roughly in a sweet spot of 50-150, although I'd be curious if Sean has additional comments here.

    It certainly makes sense, that in a "scientific survey" a sample size is going to have some degree of +/-, and so, theoretically at least, above a threshold of 5, it *shouldn't* matter, as the sample size plus or minus a few basis points, should hold true for the general population of your app --- if you're doing the survey right.

    What do you think Anuj?

    • AA

      Anuj Adhiya

      over 5 years ago #

      Thanks for that perspective David.

      I'll be honest, I'm not a statistician of any sort, but from what little I know: the larger the sample size, the more sure you can be sure that responses truly reflect the population & hence the smaller your margin of error.

      I suppose I was trying to understand whether there is a "better" range to the sample size surveyed that gives a clearer indication of having achieved (or being close to) PM fit or not.

  • RP

    Ritika Puri

    over 5 years ago #

    Are you using software that calculates a p-value or confidence interval? These metrics will help you understand whether your sample (i.e. conclusions) are representative of the general trend (i.e. that your product sucks).

    Statisticians say that an ideal sample size is an n of 30, but your p-value will tell you whether your results are statistically valid.

    The findings depend on your survey methodology too. If you're taking a quantitative approach, you need to make sure that your survey is designed for reliability and validity.

    If you're arriving at these conclusions based on qualitative research (customer interviews, conversations, etc), you should seek to uncover a pattern rather than a specific number.

    • AA

      Anuj Adhiya

      over 5 years ago #

      Thanks for that input, Ritika.
      The point about looking to uncover patterns vs specific numbers when doing qualitative research is spot on - message received.

    • SC

      Shana Carp

      over 5 years ago #

      If they do it that way, they may run into the "even Miller" problem - you can't check the results while the test is running!

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
11
11