Leave a comment

๐Ÿ’€ When crawler traps come at your door at Halloween, will they scare you?

Did you know that search engine ๐Ÿ•ท๏ธ๐Ÿ•ท๏ธ๐Ÿ•ท๏ธ sometimes really go off the rails when they hit a crawler trap? In some cases, crawler traps can even completely tank your site's rankings ๐Ÿ“‰

Crawler traps are real, and they're a threat to your SEO performance as confirmed by SEO experts such as Dawn Anderson, Patrick Stox, Dan Taylor, Paul Shapiro, David Iwanow and Sam McRoberts.

Therefor, we put together an easy-to-follow guide on how to identity, fix and avoid crawler traps.

Here's what we cover:

๐ŸŽƒ What are crawler traps?
๐ŸŽƒ Why should you worry about crawler traps?
๐ŸŽƒ How to identify crawler traps
๐ŸŽƒ Common crawler traps and how to avoid them
๐ŸŽƒ Best practices to avoid crawler traps overall

I'm curious to hear what kind of crawler traps you've come across, and how you handled them. Please drop them in the comments!

  • IG

    Igor Gorbenko

    about 1 year ago #

    Thanks for sharing. Solid read as always. Will fwd to our SEO guy :)

    • SV

      Steven van Vessum

      about 1 year ago #

      Thanks Igor, appreciate that. Let me know what your SEO guy says, and please also ask him if he's dealt with crawler traps before. I kind of "collect" them :D

  • NA

    Neil Andrew

    about 1 year ago #

    Great read as always, thanks for sharing! :)

  • JC

    Jacky Chou

    about 1 year ago #

    I get how crawler traps can be an issue for SEO, but I can imagine it's an issue for larger sites mostly. Around what page size would you say people have to worry about things like crawler traps?

    • SV

      Steven van Vessum

      about 1 year ago #

      Thanks for asking Jacky.

      As crawler traps very much bound to crawl budget, I'd go for the same size: from 10,000 pages and up. Also see: https://www.contentkingapp.com/academy/crawl-budget/.

      Please note that small sites can also suffer from crawler traps, but it's less likely and you'll spot it a lot easier simply due to the size of the site as it stands out like a sore thumb.

      • NR

        Nikola Roza

        about 1 year ago #

        That was a sigh of relief, by the way:)

        I was reading and it all seemed really complicated and technical to me; it's good that smaller sites are less exposed to this kinds of problems, and that it's easier to take care of them if they do occur.

        Also, I like your approach to solving crawl traps problems- it is better to prevent them from ever happening that to fix them later. I like that approach because it applies to all areas of business and life. Being proactive is key!

        Cheers Steven!

  • PM

    Pete McAllister

    about 1 year ago #

    I've been the victim of a crawler trap before, I didn't actually pick up on it via a screaming frog crawl though, it was actually watching Google go down the rabbit hole via a log file analysis. Any tips on picking these up for tricky crawler traps if you don't have access to a client's log files and they're not obvious from a simulated desktop crawl?

    • SV

      Steven van Vessum

      about 1 year ago #

      Thanks for sharing Pete!

      Most crawlers will typically pick up on crawler traps, but since these are all taking snapshots of your site at a given time, what you really want is to _monitor_ your site for issues and changes. It's something that is close to my heart, as the co-founder of ContentKing: what we're seeing is that small changes, be it content or technical in nature, can already create a crawler trap. Imagine this scenario:

      - At moment A you determine there are no crawler traps based on your crawl.
      - At moment B faulty links are introduced, which are picked up by search engine crawlers
      - At moment C you determine there are no crawler traps based on your crawl, because the faulty links have been removed again, so everything looks fine.

      In reality, search engine crawlers have found a thread and they pulled it, and your crawls don't reflect that.

      That's why we stress the importance of monitoring so much.

  • JM

    Joe Murfin

    about 1 year ago #

    thanks for sharing Steven, always insightful and helpful.

  • DG

    Dejan Gajsek

    about 1 year ago #

    It sounds like crawler traps can be a real issue for larger sites. Do you think they are used as an attack vector for negative SEO as well?

    • SV

      Steven van Vessum

      about 1 year ago #

      That's an interesting question, I think yes. When there are crawler traps lurking in the shadows, and someone sends links to those vulnerable sections of a site, that could impact it for sure. Crawlers will follows those links and crawl those sections.

      I'm sure people have tried it.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! Youโ€™re in.