Leave a comment
Get the GH Bookmarklet

Ask GH

We're running our first retention experiments soon and would appreciate a little help. Would you recommend changing the product and observing how cohort retention changes... or would you A/B test different versions of the product? Why?

  • AA

    Anuj Adhiya

    10 months ago #

    Its both.

    An A/B test simply tells you whether the difference in behavior between two populations is potentially meaningful or not.
    By potentially meaningful, I don't necessarily mean statistical significance, it could also be significance from a business ROI perspective.
    The reason for this distinction is that if this is the first time running a certain type of test (ie a discovery type test) you're not necessarily looking for statistical significance as much as a signal that tells you it's worth doubling down on (ie doing optimization tests). With optimization tests, you're more concerned with statistical significance because you want more surety that the changes you make are positively impacting your conversion rates (and any other metrics of interest).

    In either type of test, if you have a potentially meaningful signal, a cohort analysis of that same test will provide more granularity on this signal based on user characteristics/variables (eg acquisition channels, specific user flows within the product, day of the week, time of day, location, device type etc).
    This will help fine-tune your understanding of which user characteristics/variables lead to this positive signal in your tests. Conversely, it will also tell you what characteristics/variables don't contribute to the signal.
    This, of course, will then lead to more focused analyses for future optimization type tests.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.