Leave a comment
Get the GH Bookmarklet

Ask GH

Sean Ellis seems to use an ICE score, potential impact, confidence and how easy it is to implement. Makes sense. But Im guessing people are customizing the metrics quite a bit to fit their own needs/entrepreneurial philosophy. These are the ones I have added: 1. Personal time (Less points if I have to manage the experiment. More points if I can delegate it very easily) 2. Defence (More points if the experiment increases the competitive advantage/defence possibility of the business. Minus points if it lowers it) http://abovethecrowd.com/2011/05/24/all-revenue-is-not-created-equal-the-keys-to-the-10x-revenue-club/ Have you played around with different metrics to prioritize your own ideas?

  • SE

    Sean Ellis

    almost 4 years ago #

    Hey Sebastian, thanks for starting an interesting discussion. I really like your suggestions. Another thing that I like to consider is evidence that a test will work. For example, if you know people are having a specific conversion challenge, then a test that addresses that issue is much more likely to be successful than a random guess. You might know about the cause of a conversion issue because it came up during a user test or a survey targeted to people abandoning the funnel at that point. This could probably be considered as part of the confidence portion on the ICE score though...

    • SJ

      Sebastian Johansson

      almost 4 years ago #

      Good idea.

      The confidence score is kind of a gut feeling really. And there might be a problem if you are using the system to rank ideas based on their score and everyone adding ideas rank their ideas by their own gut feeling. Johns perception of a confidence level might be very different to optimistic Peter.

      So, one solution might be to add:

      Confidence level:
      5 - Multiple answers in survey
      .
      .
      .
      1 - Im just guessing

      This way all people ranking their ideas has are on the same level. They know what a certain score correlates from an evidence point of view.

      • PI

        Petru Iacob

        over 3 years ago #

        Agree with you both. However, we're talking about data driven decisions here. A confidence score should never be higher than 3-4 if it's just a "gut feeling". If it's based on previous tests or other data, only then should it be higher.

        Using the 3 numbers score is good (IMO) because, at least for small tests, spending too much time on evaluating the ICE score can defocus you. So while thinking of scores, you could actually go and execute on that task.

  • MB

    Morgan Brown

    almost 4 years ago #

    We have a monthly or quarterly focus area that we use as a lens to prioritize our tests around. That focus area rolls up to a core business metric we're trying to improve around the AARRRR framework.

    For example, we worked for the last few months on the conversion rate of our subscription funnel (Revenue) so in addition to the ICE score, we used the lens of our goal of improving the conversion rate of our funnel as the filter for all of the experiments we are running.

    After a few months focused here and improvement, we turned to retention and are repeating the process. We also consider things like:

    - resources available (are there tests that we can run by leveraging different resources so we don't get bottlenecked)
    - in channel experiments vs. on-site experiments
    - what else is running, where can we experiment without impacting other tests?

    Hope this helps

    • SJ

      Sebastian Johansson

      almost 4 years ago #

      "We have a monthly or quarterly focus area"

      Why did you decide focusing on one area would create more growth vs skimming the milk in all areas? Just picking the ideas with the highest scores, whatever area they belonged to.

      Do you feel it creates bottlenecks to focus on one area? Only so many ab-tests you can do on a signup page, for example, unless you have a very big amount visitors. Or unless you re prepared to wait 5 years for statistical significance.

      • MB

        Morgan Brown

        almost 4 years ago #

        Hey Sebastian,

        We pick a focus area because we only have so many resources and so much time, and we can identify the gaps in our user flow.

        So for example, I could spend time trying to grow traffic 10% or 20% in a given quarter, or I could work on improving our conversion rate 5% to 10%.

        Then it's just a math problem. Which one results in greater return since I can't do all of them at once?

        We haven't found the focus to be limiting, in fact, we've found the opposite, that when we focus on an area as a team we get better idea flow, better data analysis, higher quality test ideas, and better execution.

        Sure, we run into not being able to run as many tests as we'd like, but we have decent (not huge, sub 1M users/month) traffic, but the other benefits outweigh it.

        When we do run into a blocking a/b test, then we look at how we can use our experiment inventory to fit in some tests elsewhere, or we will do things that aren't A/B testing, but super-valuable to improving like: customer interviews, UX reviews via screen recordings, surveys, etc. That help ID opportunities for better experimentation going forward.

        Also, we set a sample size before running a test and if it is proving to be inconclusive at that sample size, we kill it and stick with the control and move on to something else. Keeps us from being frozen in limbo.

  • AB

    Andrei Baklinau

    over 2 years ago #

    How we prioritize ideas:
    1. It should be aligned with company wide goals (focus)
    2. It should be aligned with quarter goals
    3. It should have impact on business metrics (revenue, clients, AOV). If it doesn't make money, it shouldn't be prioritized.
    4. It should be easy-to-implement (without big dependence on product team / programmers, because it significantly increases time of delivery)
    5. It should be scalable and measurable, if it works.

  • HQ

    Hila Qu

    almost 4 years ago #

    This is a great question.
    Similar to your "defence" metric, I think deeper about the "impact" of the test:
    - Think about the "One thing" I improve, that will have benefit all other areas in my business
    - Think about the stage of the funnel, usually the improvement is made deeper in the funnel, the longer effect it will has
    - Think in quantitative terms about the impact: Which metric this test will impact, by how much, how this will impact my overall business metric
    - Are there other ongoing tests that will interfere with this test

    • SJ

      Sebastian Johansson

      almost 4 years ago #

      "- Think about the "One thing" I improve, that will have benefit all other areas in my business"

      Thats a great one. I heard Tim Ferris say a similar one in the context of finishing your to do lists. "Do the tasks that will make all the other tasks unnecessary to do." Or something like that.

      "- Think about the stage of the funnel, usually the improvement is made deeper in the funnel, the longer effect it will has"

      How do you mean exactly?

      • HQ

        Hila Qu

        almost 4 years ago #

        Tricks/Tactics have expiration date, so if the improvement is really at the top of funnel, for example, you changed the color of your email collector, after a month, it may not work as well. On the other hand, if the improvement is really deep in the funnel, for example, you improved a function of your product, then the effect is likely to last longer.

        Of course, this is just one aspect to consider. I learned about this idea from a podcast of Brian Balfour

        https://growthhackers.com/articles/podcast-brian-balfour-on-creating-meaningful-growth/

  • AP

    Andre Pinantoan

    over 3 years ago #

    I've been really pushing for this recently and finds ICE score to be quite comprehensive. But a lot of thinking goes into determining the impact, confidence and ease scores. A book I highly recommend reading that is related to this is "Superforecasting" - because that's what you're essentially doing. You're forecasting what an experiment might achieve without running it first.

  • JP

    Jason Peck

    almost 4 years ago #

    I wouldn't overthink it too much. Hypothesis re: impact (and how you arrive at the number/s), confidence in this, and level of effort required (dev needed? or just copy/creative needed? etc) are really what matter most.

    • SJ

      Sebastian Johansson

      almost 4 years ago #

      Yeah, Im probably going a bit asperger on this.

      But I like systems that dont require my input. A system that can be set up and handed over to employees to execute. Right know it works great if I manage it myself. But not good enough to hand over, unless I hire a growth hacker to manage it.

  • AA

    App360 App

    over 3 years ago #

    customer short term value

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
52
52