Leave a comment

The post looks at an A/B test with a simple copy change and how it improved conversion rates by 139%. The idea behind the A/B test was to give users immediate access to the product, via a recorded demo, instead of having them receive a personal phone call from our sales team later on.

  • JS

    Jordan Skole

    about 3 years ago #

    This is a great post. I'd be curious to know after you've had this implemented for a little while if the quality of leads has changed at all. My assumption would be that the DIY watch a demo customers are going to be lower price point, but those that want a consultation have a higher price point.

    Just curious if you've noticed anything so far?

    • DR

      dave rigotti

      about 3 years ago #

      Totally agree with you here. I think this is an obvious result, but having submissions / CTR as a goal for "watch a demo" is pretty weak. I'd love to see how this affected revenue or profit (including sales time to follow up and sell to lower quality leads).

      • AT

        Andrew Thompson

        about 3 years ago #

        @jordanskole & @drigotti you're completely right. It's still very early days after launching this new version and the quality of user is much more of a lagging indicator that we'll follow-up on. It will most likely get affected but we'll have to look at the overall net impact to see if this change is worth while or not.

        We have targets for both acquisition (lead capture) and activation (on-boarding), this A/B test has proved itself very effective in the acquisition stage and we are watching how this affects the activation stage as well.

  • LT

    Luke Thomas

    about 3 years ago #

    What's the sample size?

    • DV

      Dennis van der Heijden

      about 3 years ago #

      I cannot see sample size but I know Optimizely's default is 25 conversions with 100 visitors (as minimum before showing anything. I also see their looking at a weeks worth of data. Most cases are publishing only with a good confidence and this one has 100% to get that it's a lot of data...

      I like the full report, we should do more of these cases at http://www.convert.com :-)

      • AT

        Andrew Thompson

        about 3 years ago #

        @lukethomas and @dennisvdheijden unfortunately I've had to redact the actual sample size of visitors and conversions but suffice to say we ran the A/B test with a entire weeks worth of SEO, PPC and Social traffic to our GoCardless homepage (www.gocardless.com) and we far exceeded our usual 95% statistical significance threshold (Mixpanel indicated 99.9%).

        The deeper we get in the funnel, the numbers do start becoming a concern but on the homepage we have more than enough traffic to run this type of A/B test and be confident in the result.

  • AC

    Alex Cook

    about 3 years ago #

    Great post, thanks for putting it together.

    Curious if this is because potential customers perceive "watch a demo" as something they can do without going through a salesperson, whereas "request a demo" conveys something I can do on my own, right now.

    If there was previously a flow where someone talks to a human, I'd be curious to see what the post-demo-request funnel looks like. For example, what if watching a demo video gets you 4x the views but a 10x worse conversion to paid? Just thinking out loud a bit.

  • AT

    Andrew Thompson

    about 3 years ago #

    I'm happy to answer any questions about how we approached this A/B test so please ask if you're interested.

    • DL

      Dylan La Com

      about 3 years ago #

      Awesome test @andrewthompson - did you consider other variations of the test before deciding on 'Watch a demo'?

      • AT

        Andrew Thompson

        about 3 years ago #

        @dylan We did brainstorm a number of other variations like "Try the product", "Live demo", etc but we decided to test the "Watch a demo" variation first which turned out to have a great result.

  • ET

    Everette Taylor

    about 3 years ago #

    Optimizely is awesome.

  • BS

    Bhaskar Sarma

    about 3 years ago #

    Good use of active verb in the button copy. I wonder if you can bump up the conversion rate even further by making the form even simpler and eliminating the phone and payment fields

    Additionally the subheading "Find out more about ..." can be made more benefits focused and customer oriented

    • AT

      Andrew Thompson

      about 3 years ago #

      Thank you @bhas for the suggestions.

      One of the next items we want to tackle is simplifying the form fields. Given that we're in the payments space we are legally required to capture certain details at the time of sign up so we have less flexibility here than we'd like but we do have some room to improve things slightly and we'll see whether any of them have an statistically significant uptick in conversion.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
20
20