SEO Question: How Can I Prevent Spiders From Indexing Specific URLs?

I have some pages that I don’t want Google to include in their results. Is it possible to exclude certain pages so that Googlebot & other spiders don’t automatically grab them?

Deeho Says:

Quite often, you may not want to have all of your pages included in Google’s SERPs.

There are several ways to do this. The most effective is to use your robots.txt file to instruct Googlebot not to follow certain pages.

This generator tool is ideal to get you started, or use any of the following examples within your robots.txt file;

Allow indexing of all;

User-agent: *


User-agent: *
Allow: /

Disallow Indexing of Everything;

User-agent: *
Disallow: /

Disallow a Specific Folder;

User-agent: *
Disallow: /folder/

Disallow a Folder Except For Stated Files;

User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html

Robots Metatag

The Robots Metatag can be added to the header of pages that you don’t want to be indexed. Just paste the following into your header;

<meta name=”robots” content=”noindex, nofollow”>

Disallowing different robots can have several benefits, for example, many less than reputable marketing companies use bots to scrape contact information from website content. If you only allow certain reputable and beneficial bots, you can block others that are just looking for your data.

More Frequently Asked SEO Questions

Additional Reading:

What is SEO?