Leave a comment
Get the GH Bookmarklet

Ask GH

When A/B testing with client-side tools like VWO or Optimizely, flickering is when the original page content is visible for a short amount of time before the testing script changes it. There are basically two ways to deal with it:

1. Load the code asynchronously and have a JavaScript loop check for when the elements you're changing are loaded. This works great sometimes and the flicker is imperceptible. But other times, you can clearly see the Control content before it changes into the Variation. Then you no longer have a valid experiment if Variation users are consciously or subconsciously experiencing the control.

2. Load the code synchronously, blocking the page and slowing it down. Also, hide the page until it finishes loading and the changes are applied. The users see a blank page until they see the variation. Visitors bucketed into the variation never have a chance to see the control. But the variation could be perceived to be loading slower than the control, again making the experiment invalid.

We generally make choice #2 because it's difficult to explain to clients when they can see the control changing into the variation when they load a page. And we feel it creates a more valid experiment and users can't tell they are being tested on.

  • DV

    Dennis van der Heijden

    about 2 months ago #

    It depends is the proper answer as CRO-ers know.

    You are trading first paint speed for flicker called (FOOC Flash of Original Content) your options are really only:

    -1- Hide the body x-time, this allows the experiment to have enough time to make the swap of content. The downside of this approach is that your content is not visible and this affects FCP or First Contentful Paint and this becomes more important now Google Ranking includes this as signal now/soon.

    -2- Going Server Side or Hybrid. Most tools (including VWO, Optimizely and Convert.com (where I work) offer server side options. This is faster and does not have the FOOC. Sitespect offers a hybrid solution that works by modifying the code before presented (handy for JS modification) and that offers a CNAME / host name redirect. That’s also pretty fast and does not show FOOC.

    -3- You listen to the DOM-load ... Convert.com allow you to do this (again, I work there, so ‘preach’). This method offers the method where the A/B testing tool ‘listens’ to DOM load and each 50ms replaces elements that are in need of replacing. This allows a very fast first render times. See the research done by CXL on this topic a while back https://cxl.com/blog/testing-tools-site-speed/

    -4- Mode code up high in the head tag for faster loading (yes... you’d have to fight with IT for this) but worth doing.

    -5- Rewrite the terrible code the visual editor gives you (remember... I still work for Convert.com) but our and other tools give you a terrible inefficient JS code if you use the visual editor. There is no standard of coding a site and browser make things look good independent of how its coded. But a nightmare for tools like us to work with. So bad code and repetitive code might be the result. So code might be repetitive when you modify the same area multiple times (like oh move the text, then change the color, then x... 3 times same area touched, 3x the selector and change that you should manually change (or better code them up manually)

    -6- Combine methods 1 and 3 like we do by default where we are pretty sure we are all done with DOM changes in 1.2 seconds. We hide the body for 1.2 seconds and use DOM-manipulation by recurring checks 50ms. This is the performance we rolled out standard to all customers in each project by default.

    So I hope this helps you get on your way solving this issue and all the ones that are still looking for this so many years later.

  • PL

    Peep Laja

    almost 7 years ago #

    Other tips:

    1) Optimize your site for speed
    Make sure everything, especially the images load fast. Move your site onto a fast web host. Use compression, optimized images and content delivery networks.
    2) You should load the testing tool tags in the head (</head), move code higher up
    3) Take your A/B testing tool script out of the Tag Manager (if you use one, like Google Tag Manager).
    4) If you can, do split URL testing instead
    5) Pay attention to the html elements you change. If you made changes on the page itself (as opposed to doing split URL testing), the key thing to keep in mind is the amount of code that was changed. If you changed thousands of lines, you’re far more likely to see a flicker effect as opposed to changing button copy. So the scope of changes influences everything.

    • AJ

      anurag jain

      about 4 years ago #

      We are currently calling Test and Target script by DTM, if we take it out of DTM and put directly in the page code, then should we put it above or below DTM script? And is there any problem that would arise due to this solution(in data capture) or is there any downside to this?

  • PL

    Peep Laja

    almost 7 years ago #

    If you use Optimizely, you can force it to not display anything before Variation is loaded.

    The easiest to implement is some markup that forces Optimizely to evaluate variation code sooner. You can wrap your variation code as follows:

    /*_optimizely_evaluate=force*/
    //VARIATION_CODE_TO_SWAP_IMAGES_HERE
    /*_optimizely_evaluate=safe*/

    By default, Optimizely follows this algorithm for executing variation code. The above trump that algorithm and force the variation code to run as soon as Optimizely begins to load. Keep in mind that if the elements do not exist on the page by the time this code runs, the code will have no elements to execute on. This article will provide further detail.

    A second possible solution would be to hide the body of the page until all of the changes have been made. Instead of seeing the old image and then the new one, the user would not see that page at all until all the variation code has executed. Most users feel this is preferable to flashing two images. It would look like this:

    $("body").css({"visibility":"hidden"});
    //VARIATION_CODE_TO_SWAP_IMAGES_HERE
    $("body").css({"visibility":"visible"});

    • KD

      Karolis Dzeja

      almost 7 years ago #

      I believe that naively using the _optimizely_evaluate=force will create a race condition between the Optimizely library loading the DOM element loading.

      If you're loading Optimizely's library asynchronously and you're using $(“body”).css({“visibility”:”hidden”}); to hide the body. You can still get flickering. If the DOM starts loading before the library finishes loading, there will be a flicker. The way to make sure that flicker won't happen is have the CSS to hide the body load synchronously in the head before the body starts loading.

  • AB

    Andy Beard

    almost 7 years ago #

    The best way I have found without using lots of complicated frameworks for load order of javascript.

    1.Hard code the page as hidden
    2.Have the reveal loaded by the split testing code

    Some extra things to watch out for

    a) Every user will experience a different load order depending on existing cache and connection
    b) Be mindful of your other analytics and firing order too

  • PC

    Paras Chopra

    almost 7 years ago #

    Generally with VWO's asynchronous code, there should not be any flickering. In edge cases or in cases where you haven't put asynchronous code in the head of the page, flickering might happen. We've made our asynchronous code in such a way that the original page is hidden until the settings are loaded. (The page gets loaded in parallel and that's why it is async but it's not displayed).

    Probably you might see a flicker if any other code on the page unhides the page that VWO hides by default. For specific tests, we're happy to help. Get in touch with us at support@wingify.com

    With regards to VWO (or any other tool's) synchronous code, I wouldn't recommend it generally unless there are specific cases that cannot be fixed. Synchronous slows down the page speed.

  • DV

    Dennis van der Heijden

    over 6 years ago #

    Hi Karolis, we solved this flicker with Smart Insert (TM) a patent pending technology that starts making the changes as soon as the code loads (before DOM-ready) and this allows you to stop the flicker in most cases. It's part of all A/B testing software accounts on Convert.com

  • SC

    Shana Carp

    almost 7 years ago #

    Some of this has to do with latency - much of that can be solved with what Peep is saying

  • MB

    Morgan Brown

    almost 7 years ago #

    For Adobe Target, using Javascript in your mBox can get rid of flickering: https://www.youtube.com/watch?v=hfLdPVihicg

  • JB

    Justin Bougher

    almost 7 years ago #

    The problem is using a tag with testing. You are always going to be forced to make this horrible trade-off between latency and flicker. If you are stuck using that testing platform, then prioritizing flicker removal over latency is the right call.

    I am definitely bias, since I am at SiteSpect which is an A/B testing solution that doesn't use tags because of this issue.

  • ST

    Scotty Truong

    over 5 years ago #

    This is exactly we struggled with the most when running A/B tests. Sometimes, the Javascript would even broke the pages. So, we switched to only running split URL tests.

    (Shameless plug) That's also the reason why we created http://3MinuteOptimizer.com to help small business set up split URL A/B tests fast.

  • MS

    Mark Steve

    almost 5 years ago #

    Flicker is the short flash of un-styled or unchanged content before any html/javascript/css in the test take place.

    There are several techniques to make this flash as minimal as possible. The first few involve minimizing the load time of a test i.e. by avoiding using external libraries such as jQuery in your code, so you don't have to wait for it to be there. Similarly, there is no need to wait for the whole document to be ready if you are only manipulating a few elements, you can just poll for them instead.

SHARE
14
14