Leave a comment
Get the GH Bookmarklet

Ask GH

When A/B testing with client-side tools like VWO or Optimizely, flickering is when the original page content is visible for a short amount of time before the testing script changes it. There are basically two ways to deal with it:

1. Load the code asynchronously and have a JavaScript loop check for when the elements you're changing are loaded. This works great sometimes and the flicker is imperceptible. But other times, you can clearly see the Control content before it changes into the Variation. Then you no longer have a valid experiment if Variation users are consciously or subconsciously experiencing the control.

2. Load the code synchronously, blocking the page and slowing it down. Also, hide the page until it finishes loading and the changes are applied. The users see a blank page until they see the variation. Visitors bucketed into the variation never have a chance to see the control. But the variation could be perceived to be loading slower than the control, again making the experiment invalid.

We generally make choice #2 because it's difficult to explain to clients when they can see the control changing into the variation when they load a page. And we feel it creates a more valid experiment and users can't tell they are being tested on.

  • PL

    Peep Laja

    over 4 years ago #

    Other tips:

    1) Optimize your site for speed
    Make sure everything, especially the images load fast. Move your site onto a fast web host. Use compression, optimized images and content delivery networks.
    2) You should load the testing tool tags in the head (</head), move code higher up
    3) Take your A/B testing tool script out of the Tag Manager (if you use one, like Google Tag Manager).
    4) If you can, do split URL testing instead
    5) Pay attention to the html elements you change. If you made changes on the page itself (as opposed to doing split URL testing), the key thing to keep in mind is the amount of code that was changed. If you changed thousands of lines, you’re far more likely to see a flicker effect as opposed to changing button copy. So the scope of changes influences everything.

    • AJ

      anurag jain

      almost 2 years ago #

      We are currently calling Test and Target script by DTM, if we take it out of DTM and put directly in the page code, then should we put it above or below DTM script? And is there any problem that would arise due to this solution(in data capture) or is there any downside to this?

  • PL

    Peep Laja

    over 4 years ago #

    If you use Optimizely, you can force it to not display anything before Variation is loaded.

    The easiest to implement is some markup that forces Optimizely to evaluate variation code sooner. You can wrap your variation code as follows:

    /*_optimizely_evaluate=force*/
    //VARIATION_CODE_TO_SWAP_IMAGES_HERE
    /*_optimizely_evaluate=safe*/

    By default, Optimizely follows this algorithm for executing variation code. The above trump that algorithm and force the variation code to run as soon as Optimizely begins to load. Keep in mind that if the elements do not exist on the page by the time this code runs, the code will have no elements to execute on. This article will provide further detail.

    A second possible solution would be to hide the body of the page until all of the changes have been made. Instead of seeing the old image and then the new one, the user would not see that page at all until all the variation code has executed. Most users feel this is preferable to flashing two images. It would look like this:

    $("body").css({"visibility":"hidden"});
    //VARIATION_CODE_TO_SWAP_IMAGES_HERE
    $("body").css({"visibility":"visible"});

    • KD

      Karolis Dzeja

      over 4 years ago #

      I believe that naively using the _optimizely_evaluate=force will create a race condition between the Optimizely library loading the DOM element loading.

      If you're loading Optimizely's library asynchronously and you're using $(“body”).css({“visibility”:”hidden”}); to hide the body. You can still get flickering. If the DOM starts loading before the library finishes loading, there will be a flicker. The way to make sure that flicker won't happen is have the CSS to hide the body load synchronously in the head before the body starts loading.

  • AB

    Andy Beard

    over 4 years ago #

    The best way I have found without using lots of complicated frameworks for load order of javascript.

    1.Hard code the page as hidden
    2.Have the reveal loaded by the split testing code

    Some extra things to watch out for

    a) Every user will experience a different load order depending on existing cache and connection
    b) Be mindful of your other analytics and firing order too

  • PC

    Paras Chopra

    over 4 years ago #

    Generally with VWO's asynchronous code, there should not be any flickering. In edge cases or in cases where you haven't put asynchronous code in the head of the page, flickering might happen. We've made our asynchronous code in such a way that the original page is hidden until the settings are loaded. (The page gets loaded in parallel and that's why it is async but it's not displayed).

    Probably you might see a flicker if any other code on the page unhides the page that VWO hides by default. For specific tests, we're happy to help. Get in touch with us at support@wingify.com

    With regards to VWO (or any other tool's) synchronous code, I wouldn't recommend it generally unless there are specific cases that cannot be fixed. Synchronous slows down the page speed.

  • SC

    Shana Carp

    over 4 years ago #

    Some of this has to do with latency - much of that can be solved with what Peep is saying

  • MB

    Morgan Brown

    over 4 years ago #

    For Adobe Target, using Javascript in your mBox can get rid of flickering: https://www.youtube.com/watch?v=hfLdPVihicg

  • JB

    Justin Bougher

    over 4 years ago #

    The problem is using a tag with testing. You are always going to be forced to make this horrible trade-off between latency and flicker. If you are stuck using that testing platform, then prioritizing flicker removal over latency is the right call.

    I am definitely bias, since I am at SiteSpect which is an A/B testing solution that doesn't use tags because of this issue.

  • DV

    Dennis van der Heijden

    over 4 years ago #

    Hi Karolis, we solved this flicker with Smart Insert (TM) a patent pending technology that starts making the changes as soon as the code loads (before DOM-ready) and this allows you to stop the flicker in most cases. It's part of all A/B testing software accounts on Convert.com

  • ST

    Scotty Truong

    over 3 years ago #

    This is exactly we struggled with the most when running A/B tests. Sometimes, the Javascript would even broke the pages. So, we switched to only running split URL tests.

    (Shameless plug) That's also the reason why we created http://3MinuteOptimizer.com to help small business set up split URL A/B tests fast.

  • MS

    Mark Steve

    almost 3 years ago #

    Flicker is the short flash of un-styled or unchanged content before any html/javascript/css in the test take place.

    There are several techniques to make this flash as minimal as possible. The first few involve minimizing the load time of a test i.e. by avoiding using external libraries such as jQuery in your code, so you don't have to wait for it to be there. Similarly, there is no need to wait for the whole document to be ready if you are only manipulating a few elements, you can just poll for them instead.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
9
9