Use robots.txt


If you have a section of your site with pages that move around or get added and deleted a lot, consider blocking search engines from indexing them in their databases. You definitely shouldn’t use META tags on pages that move a lot, either.

Create a file called robots.txt in the root directory (the one you refer to as ‘/’). Follow this format:

# robots.txt for
User-agent: *
Disallow: /disappearing/ # These pages move a lot.
Disallow: /soontobe404.htm


‘User-agent: *’ means that it applies to all search engines. Anything preceded by a ‘#’ is simply a note to yourself. You can block entire directories or individual files using robots.txt. Use it!