Question: What Are Crawlers / Search Engine Spiders?

I have read about Google using crawlers sometimes called spiders or bots…. what are they?

Deeho Says:

Search engines do indeed your ‘spiders’ to ‘crawl’ website content. Think of it like this, tiny little robots that visit your site, and then follow the first 100 links on each page, taking notes and sending everything they find back to Google HQ.

Once the information has been gathered by GoogleBot, the information gets fed through Googles algorithmic filters and you will then be placed somewhere in their results based on the density of keywords found on your pages and the relevance of your content and back-links pointing to your site.

The frequency of visits to your site by search engine spiders will depend on several factors including the frequency of updates to your site, visitor volumes, etc.

If Google Bot doesn’t find new content when they visit then it will come less often, but if it finds new content on every visit then it will visit more often.

GoogleBot will spider a site anywhere from every few hours to once every month or so…… even longer in some cases. If you get in the habit of publishing new content on a regular basis then it will encourage more frequent visits from GoogleBot & other search engine spiders.

Additional Reading:

What is SEO?