Leave a comment

Crawl budget is the attention search engines give your website.

Google says SEOs shouldn't obsess about crawl budget, because they'll figure out how often to crawl sites. While that sounds nice in theory, it's better to be in control and guide search engines when crawling their sites. Especially when we're talking about larger websites (10,000+ pages).

In this article we'll cover:

  • What crawl budget really is
  • Why you should care about crawl budget
  • How you can get a feel for your site's crawl budget
  • How to optimize your crawl budget
  • How to increase your crawl budget

As a BONUS, leading SEOs such as Barry Adams, Paul Shapiro, Cyrus Shepard, Jenny Halasz, Dawn Anderson, Kevin Indig and Ross Tavendale are sharing their best practices and experiences around crawl budget.

  • JC

    Jacky Chou

    over 1 year ago #

    People have been going nuts about crawl budget, but Google says you don’t really have to worry about it because they’ll figure it out on their own. What’s your opinion on this, at what point should you be worrying about crawl budget?

    • SV

      Steven van Vessum

      over 1 year ago #

      That's a very common question around crawl budget (see https://twitter.com/Erik_Episcopo/status/1032307917226209280 for instance too).

      In general I'd say you have to work on crawl budget optimization when:
      - your platform contains 10,000 pages or more.
      - you're seeing that pages aren't getting crawled as much as you'd like by search engines.
      - you're seeing your crawl stats in Google Search Console / Bing Webmaster Tools go mental.

  • VF

    vladimir fefer

    over 1 year ago #

    I really like the technical SEO stuff Contentking has been publishing lately.

  • IG

    Igor Gorbenko

    over 1 year ago #

    When I was learning SEO I was told that the ultimate goal of technical SEO is to increase your crawl budget or make google visit your sire more frequent and scan more pages.

    That was some years ago but still has some truth to it.

    • SV

      Steven van Vessum

      over 1 year ago #

      Thanks for chiming in Igor!

      Yep, the fundamentals haven't changed. Sure, SEO develops fast and there's more factors to take into account but at the end of the day it comes down whether or not pages deserve to be crawled, and what your server can handle in terms of crawling.

  • DG

    Dejan Gajsek

    over 1 year ago #

    As I remember early at the start of the year, there was a popular strategy (which first appeared on Brian Dean's Backlinko) to get rid of the old content that doesn't receive any views. The thinking behind it was to just keep the content in small number and focus on the quality of it (which makes perfect sense)

    In your opinion, where is the line (or at what condition) would you keep thousands of pages under one domain (would it be traffic, bounce rates per page) apart from just getting rid of dead weight?

    • SV

      Steven van Vessum

      over 1 year ago #

      Thanks for your comment Dejan. I think you're referring to "content pruning".

      Here's my take on this: content pruning and crawl budget are very much intertwined. You don't want search engine crawlers to spend time on next to useless pages, so you cut away crappy ones and focus on improving the ones that have potential.

      I would happily keep thousands of pages on a site, as long as they are:
      - of good quality (or they soon will be)
      - serving a clear purpose (think beyond SEO)
      - are ranking for queries (or if not, they should at least be part of your master plan)
      - are sending conversions

      If your pages don't check any of these boxes, get rid of them.

  • PM

    Pete McAllister

    over 1 year ago #

    Another great piece from Content King. I saw you replied to another comment saying that you should only worry (as a rule of thumb) about crawl budget when your platform has around 10k pages or more.

    One issue I've come across with a couple of sites is that they think they only have a couple of hundred pages or so. But there's what I call a 'worm hole' that the SEO or marketer doesn't know about which expands that base of a couple hundred pages into thousands and thousands.

    So it's definitely worth running a crawl to check even if you think you've only got a small amount of pages!

    • SV

      Steven van Vessum

      over 1 year ago #

      Great point Pete, that's a common misconception indeed.

      Often people think they have way less pages than they do in reality. At ContentKing, monitoring pages being added, removed, redirected and changed is a core part of what we do. So you're preaching to the choir - you should definitely have systems in place to track how many pages you have on your site, ideally in a monitoring set up rather than doing one-offs. That way you can back-trace events.

      We wrote an article on these 'worm holes' you mention, if you're interested in reading more on this check out: https://www.contentkingapp.com/academy/crawler-traps/.

  • NR

    Nikola Roza

    over 1 year ago #

    Hi Steven, thank for his guide.

    I read and it was interesting, but when I reached the part where you say that only someone with 10 000+ page website should worry about crawl budget- I almost gave up.

    I said ALMOST.

    I kept reading, because:

    a) I understand the point of efficient crawling, even for tiny sites like mine
    b) I intend to grow my website over time so... it's best to learn now and re-read when needed:)

    I do have a question for you.

    I noticed that fetch as Google is weird at times. For example, I update my page significantly and resubmit it with fetch as Google. and all goes well, the new stuff gets indexed.

    But then I remember I made some typos, so I go back and fix them. I also add a sentence or two or new content.
    Then I go and resubmit but this time- nothing happens.
    It's as if Google knows I made a minor change and can't be bothered to index right away.

    Tell me, is that a thing or am I imagining things?


    • SV

      Steven van Vessum

      over 1 year ago #

      Glad you kept on reading :)

      As for your question about "Fetch as Google" - I'm not aware this is a though specifically for "Fetch as Google". You could be seeing things that aren't there :)

      You could run an experiment on this to see if you can find a clear pattern. That would make for interesting content!

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.