Leave a comment

πŸ’€ When crawler traps come at your door at Halloween, will they scare you?

Did you know that search engine πŸ•·οΈπŸ•·οΈπŸ•·οΈ sometimes really go off the rails when they hit a crawler trap? In some cases, crawler traps can even completely tank your site's rankings πŸ“‰

Crawler traps are real, and they're a threat to your SEO performance as confirmed by SEO experts such as Dawn Anderson, Patrick Stox, Dan Taylor, Paul Shapiro, David Iwanow and Sam McRoberts.

Therefor, we put together an easy-to-follow guide on how to identity, fix and avoid crawler traps.

Here's what we cover:

πŸŽƒ What are crawler traps?
πŸŽƒ Why should you worry about crawler traps?
πŸŽƒ How to identify crawler traps
πŸŽƒ Common crawler traps and how to avoid them
πŸŽƒ Best practices to avoid crawler traps overall

I'm curious to hear what kind of crawler traps you've come across, and how you handled them. Please drop them in the comments!

  • IG

    Igor Gorbenko

    9 months ago #

    Thanks for sharing. Solid read as always. Will fwd to our SEO guy :)

    • SV

      Steven van Vessum

      9 months ago #

      Thanks Igor, appreciate that. Let me know what your SEO guy says, and please also ask him if he's dealt with crawler traps before. I kind of "collect" them :D

  • NA

    Neil Andrew

    9 months ago #

    Great read as always, thanks for sharing! :)

  • JC

    Jacky Chou

    9 months ago #

    I get how crawler traps can be an issue for SEO, but I can imagine it's an issue for larger sites mostly. Around what page size would you say people have to worry about things like crawler traps?

    • SV

      Steven van Vessum

      9 months ago #

      Thanks for asking Jacky.

      As crawler traps very much bound to crawl budget, I'd go for the same size: from 10,000 pages and up. Also see: https://www.contentkingapp.com/academy/crawl-budget/.

      Please note that small sites can also suffer from crawler traps, but it's less likely and you'll spot it a lot easier simply due to the size of the site as it stands out like a sore thumb.

      • NR

        Nikola Roza

        9 months ago #

        Phew!
        That was a sigh of relief, by the way:)

        I was reading and it all seemed really complicated and technical to me; it's good that smaller sites are less exposed to this kinds of problems, and that it's easier to take care of them if they do occur.

        Also, I like your approach to solving crawl traps problems- it is better to prevent them from ever happening that to fix them later. I like that approach because it applies to all areas of business and life. Being proactive is key!

        Cheers Steven!

  • PM

    Pete McAllister

    9 months ago #

    I've been the victim of a crawler trap before, I didn't actually pick up on it via a screaming frog crawl though, it was actually watching Google go down the rabbit hole via a log file analysis. Any tips on picking these up for tricky crawler traps if you don't have access to a client's log files and they're not obvious from a simulated desktop crawl?

    • SV

      Steven van Vessum

      9 months ago #

      Thanks for sharing Pete!

      Most crawlers will typically pick up on crawler traps, but since these are all taking snapshots of your site at a given time, what you really want is to _monitor_ your site for issues and changes. It's something that is close to my heart, as the co-founder of ContentKing: what we're seeing is that small changes, be it content or technical in nature, can already create a crawler trap. Imagine this scenario:

      - At moment A you determine there are no crawler traps based on your crawl.
      - At moment B faulty links are introduced, which are picked up by search engine crawlers
      - At moment C you determine there are no crawler traps based on your crawl, because the faulty links have been removed again, so everything looks fine.

      In reality, search engine crawlers have found a thread and they pulled it, and your crawls don't reflect that.

      That's why we stress the importance of monitoring so much.

  • JM

    Joe Murfin

    9 months ago #

    thanks for sharing Steven, always insightful and helpful.

  • DG

    Dejan Gajsek

    9 months ago #

    It sounds like crawler traps can be a real issue for larger sites. Do you think they are used as an attack vector for negative SEO as well?

    • SV

      Steven van Vessum

      9 months ago #

      That's an interesting question, I think yes. When there are crawler traps lurking in the shadows, and someone sends links to those vulnerable sections of a site, that could impact it for sure. Crawlers will follows those links and crawl those sections.

      I'm sure people have tried it.

Join over 70,000 growth pros from companies like Uber, Pinterest & Twitter

Get Weekly Top Posts
High five! You’re in.
SHARE
23
23