Leave a comment
Get the GH Bookmarklet

Ask GH

High Tempo Test is impressive and means a lot for the early stage startups like us. One thing I'm confused is about the balance between growth tests and product roadmap. For example: our product is a H5 platform to help user find the best hairdresser in city and make order online. On the product roadmap, we need to develop iOS/Android app to improve the user experience. This can't be done within one-week-sprint(or test). Since we are having more-than-enough tests in the idea backlogs, how to balance the tests and the roadmap? Reduce the times of tests so we can free some resource on roadmap development? PS. I try to connect the roadmap features with growth, all in all, everything we are doing to to improve the growth. But some feature can't be done in a week, which looks like a bad test since it takes too long to implement and check the result.

  • SE

    Sean Ellis

    over 5 years ago #

    Good question Johnny. Balancing resources between growth tests and core product development is a challenge for everyone. And I agree that ideally most product initiatives are expected to have a positive impact on growth (hard to justify doing them otherwise). I personally would designate that Android/iOS test as a core product development initiative and not count it as one of my weekly tests.

    • JL

      Johnny Lee

      over 5 years ago #

      Thanks for you answer Sean. Your suggestion is a typical way that many company follows, which is divide the resources into to two teams: one is focus on growth hacking while the other one is focus on core product. It's a challenge for early stage startups due to the limited resources. It also might introduce a diverse in the whole team: one part of people is focusing on growth based on high tempo tests, the other part of people is focusing on product roadmap based on the experience (how others did in the past) and believe (I believe this feature/direction is right because blablabla). When the resources have conflicts there might be a problem since these two group of people are not in line with each other. I try to let all people focus on the growth, but allow there are features or tests have more time to implement and test the result. This kind of big-test is limited during the timeline, for example we can only afford one big-test all the time and big-test should take only 4-6 weeks to get the result as a test. Still, the big-test should focus on one segment of the growth model.

      • MK

        Masha Krol

        over 5 years ago #

        This is something I've been pondering, too, thanks for bringing this up @johnnylee194 - because we are only 4 currently, we have resolved this problem by literally having growth be the North Star that drives our roadmap. I appreciate that this may not scale as we grow the company, but as Sean says above, it _should_ be difficult to justify doing things that wouldn't impact growth at all, because, why? :)

        I think as we grow the company, we may want to split groups out based on focus areas, and still have everyone be on a growth team.

        I also hear you on some experiments taking longer than a week to run - especially in the initial stages when we need to wait longer than a week to get enough data, it can be a little unnerving to have tests sitting in Active/Running for more than one growth meeting in a row. But, this makes more sense to me than being dogmatic about moving things in and out every week even without having acquired enough data. I think this is what you're calling big-tests - I like the term! :)

  • PV

    Philip Verghese Ariel

    over 5 years ago #

    Hi Johnny, Thanks for sharing this valuable information.
    May you have a profitable day.
    P V Ariel

  • RM

    Rhys Mohun

    over 5 years ago #

    Hey Johnny, great question to ask at any stage of growth. You're right, balancing resources around running tests is a challenge because we're under pressure to deliver features, user experience, the product roadmap, etc.

    A great testing program should be as flexible as possible to allow you to test against a variety of measures. Our own team at Roadmunk will run small usability tests alongside feature development, and those tests can range from weeks long to months long. We just have to be diligent about timing. We look well into the future when planning out tests. An example testing schedule that would run in tandem with your roadmap plans may look like this (this isn't real data, drafted this for a customer support case):


    I always like to remind marketers and product people that you have to be ready to take action after tests. If you learn from an experiment it's critical that you take action right away, otherwise you've missed the opportunity to capitalize on the time spent. Not only that, but I believe teams should have a pre-determined set of activities for any test. If challenger A wins the test, we will take action X. Don't leave it open to debate.

    Best of luck! Drop by Roadmunk's chat if you need any help setting up a testing roadmap.