Leave a comment

Recently we were working with some clients, helping them look at their growth using marketing analytics. They all had revenue, and wanted to grow faster. They knew they could do better but needed help with these questions: "where do I start?" "what tactics should I use?" "how should I measure my progress?" "where should I focus?" Maybe that seems familiar to you? Well,ou need a strategy. Not just any strategy, but a disciplined application of marketing experiments.

Here's the step-by-step strategy we used, and you can use this very same strategy today.

  • PC

    Patrick Campbell

    over 4 years ago #

    *Love* this post, but I think my absolute favorite part is the @sean growth hack lean.

  • CP

    Connor Phillips

    over 4 years ago #

    Great share @paulmboyce! I think the following points really stuck out to me and might be helpful for the TL;DR audience:

    Tactics = What you will test
    Strategy = How you organize your work

    Strong point : Think in percentages when you look at each stage of pirate metrics. Major point to consider before the creative process.

    -Focus and prioritize your top three experiments. I think a lot of marketing teams breakdown when they reach this point. They come up with hundreds of great ideas to test and then attempt test all at the same time and end up spreading their resources very thin with little to no actionable items after the experiments end.

    Strong Point 2 : Write a hypothesis for each experiment.
    -if we [action] our [metric or metrics] will increase by [goal/kpi percentage]

    Strong Point 3: Establish a baseline to understand what these changes mean.

    -Proper measurement is the key to understanding experiment results.

    -Once you find a clear winner, don't be content, keep improving that element or process.

    • BL

      Brian Lang

      over 4 years ago #

      "Proper measurement is the key to understanding experiment results."
      Agreed, but unfortunately this article doesn't go into much detail on how to do this properly.

      • PM

        Paul M Boyce

        over 4 years ago #

        Yes, good feedback Brian. The article is already pretty meaty, but would a follow-on article around measurement be helpful to you? What metrics are you typically watching to make decisions?

        cheers, Paul

    • PM

      Paul M Boyce

      over 4 years ago #

      Hi Conor, really glad you got such good key points from the article! I LOVE how you've summarised the key takeaways!! Great job. Yes, totally agree, for actionable results, it takes the discipline to test cleanly and be clear on learning outcomes, otherwise it's just too easy to be spinning your wheels.

      Think I should make a summary up front for the TL;DRs?

      thanks again!
      Paul

      • CP

        Connor Phillips

        over 4 years ago #

        I think the TL;DR has become a standard in blogging practice, but I think it comes down to your sites data. What is the avg. time on page for your blog articles? What's the bounce rate? Do you have a mechanism in place to track the percentage of the page users have scrolled through? These metrics and measurements should be good guides into seeing if there is a need for the TL;DR to capture that audience of users who aren't engaged due to the length of the article. Just my input, I'm sure other people approach this problem differently, but I like to look at as many data points as possible :)

        • PM

          Paul M Boyce

          over 4 years ago #

          Good suggestions. I like it Connor. I'll go dig into the data a little. I can add the TL;DR and measure the difference on bounce rate/time on page in Google Analytics. Not currently measuring scrolls (actually this is something on our Product Backlog for http://popcornmetrics.com!) - its not just users asking for that - I want it too!! ;)

          cheers,
          Paul

    • RG

      Robert Graham

      over 4 years ago #

      Thanks for the article @paulmboyce.

      One point to consider in measuring the stages of a funnel is segmentation. The larger your funnel gets, the more key this becomes to building successful experiments.

      The reason is http://en.wikipedia.org/wiki/Simpson%27s_paradox. Understanding your funnels by segment is incredibly powerful.

      • PM

        Paul M Boyce

        over 4 years ago #

        Hey @rgraham - very interesting that Simpson Paradox.
        Leads me to two questions:
        (1) Do you tend to break your large funnel into smaller "sub-funnels" when you're focusing in on a key area?

        • RG

          Robert Graham

          over 4 years ago #

          This depends on your hypothesis. You may run certain experiments explicitly for a specific segment, you may run an experiment to improve signups that overlaps all segments, or you might simply not know what your segments are or should be.

          The experiments for specific segments should fit nicely into their own funnel.

          Global experiments are trickier, but the simplest approach is to maintain an overall percentage conversion rate for each step as well as a per segment conversion rate for each step. You don't want a 10% improvement in overall signups that alienates your highest value segment.

          Some segments SaaS should keep in mind:

          1. monthly signup cohorts
          2. 0-90 day lifetime users (churn for early users looks nothing like churn for long-timers)
          3. activation level cohorts
          4. plan level cohorts
          5. business level segments (perhaps you sell to enterprise and startups, or universities and corporate educators)

          • PM

            Paul M Boyce

            over 4 years ago #

            Yes it was the global experiments I was thinking of - they are definitely trickier. I like the segmentation ideas you've suggested, especially the segmentation by plan. Especially interesting is then how even early funnel stage tests (channel, signups, activation) can impact not just the immediate conversation rate (at that funnel stage), but also on subsequent plan conversion & retention.

      • PM

        Paul M Boyce

        over 4 years ago #

        And (2) Curious to know how you segment your users inside a funnel during an experiment? (how do you segment, what tools do you use?).

        cheers,
        Paul

        • RG

          Robert Graham

          over 4 years ago #

          I have used a few strategies here.

          You can add properties to the events you send to Segment, KISSmetrics, Mixpanel, etc. Most of those tools support funnels that respect event properties. In the worst case you can use the API or a data export.

          Often you only need to sanity check that your rates are in line with current expectations, but some hypotheses may be looking for spikes in a particular segment.

          You can also use a backend tool like Split (ruby gem) to perform your tests. Split has a callback for when someone first receives an a/b variation and when they complete the test. You can use those hooks to store richer data locally (DB) and tie that to specific accounts.

          • PM

            Paul M Boyce

            over 4 years ago #

            Hey Robert, yes adding properties to events for sure allows that for funnels, I'd like to do the same by user profile properties. Split? Ah interesting - we use Node.js/Express and I've just spotted there's a nice AB Test package (https://www.npmjs.com/package/express-ab) that looks like it can provide a consistent experience in app (as a bonus) looks like it plays well with Google Experiments. Normally we try to use external tools, so our coding is only for building product, but there are definitely tricky cases (one is current) where we've needed to bake it into the code.

  • SE

    Sean Ellis

    over 4 years ago #

    Great post!

    • PM

      Paul M Boyce

      over 4 years ago #

      @sean - Hey Sean - OMG - I am **super honoured** you liked the post! Anything in particular that jumped out for you?

      cheers, Paul

  • PA

    Petko Anchev

    over 4 years ago #

    I appreciated the suggestion for hypothesis writing for the different experiments. I miss out on setting quntative goals/expectations for smaller projects I take on, having adopted more of a generic “it works” vs. “it doesn’t work” frame of thinking.

  • DG

    Dr. Gil Lederman

    over 4 years ago #

    Great read thanks for sharing!

  • PP

    Philippe Platteau

    over 4 years ago #

    Great post. That exactly what's happening everyday for new startup or not. Owner need many times objective views from others to go further or example toget inspiration back. +1.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
68
68