What is robots.txt

I’ll let Wikipedia explain:

The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. The information specifying the parts that should not be accessed is specified in a file called robots.txt in the top-level directory of the website.

It is normally just found in your root directory, for example http://domain.com/robots.txt. Most robots will read it – some of the spammy robots or nasty ones won’t, and there isn’t much you can do about it (not related to robots.txt, anyway). You could use a htaccess or a script to block nasty bots, but that isn’t for this article.

How to block all robots from all of your pages

This is quite simple, however at this basic level I doubt you would want to use it much

User-agent: *
Disallow: /

User-agent: * includes all user agents (* is a wildcard – standing for anything, any length). There are many pages on the internet which will help you, such as this one with a massive list of user agents.

User agent info:
Some common search engine user agents that you may be interested in are as follows:

• Main web search (some also image search):

• Google: googlebot
• MSN: msnbot
• Yahoo: yahoo-slurp
• Ask: teoma
• Alexa: ia_archiver

• Google: googlebot-image
• MSN Pic search: psbot

So if you wanted to block just Alexa from all directories, you would use:

User-agent: ia_archiver
Disallow: /

How to block a certain directory (or more than one) using robots.txt

This isn’t very different to the previous example. This will block all user agents to /private/ and /cgi-bin/

User-agent: *
Disallow: /cgi-bin/
Disallow: /private/

Comments in robots.txt

Comments in robots.txt are simply written after a “#” sign, for example:

User-agent: * #this is a comment here
# another comment
#
# another one! robots.txt is so much fun…
Disallow: /temp/

Other things to add to robots.txt

There has been a Extended Standard for Robot Exclusion proposed, adding features such as request rate (# of requests per # of seconds) and visit time (time that the bot should visit, such as 0400-0800). Some bots will follow it, but you can’t count on it (at the moment, anyway) being too handy. Here is a basic example of it in action.

User-agent: *
Disallow: /temp/
Request-rate: 1/3
Visit-time: 0400-0800

Further Reading/Related Links

• Robots.txt Generator – a neat robots.txt generator
• Robots.txt blog on Webmaster world. Making very good use of its comments feature…

This post has the “low down” on them and how to save money with them.

Even at their cheapest levels you get:

• Unlimited hosted domains – Very handy to have from a shared host
• Unlimited sub domains
• 20GB disk space (upto over 80GB, and in some cases constantly upgrading a few GB a week)
• Minimum of 1TB (1,000,000 MB) transfer (bandwidth), upto over 2.2TB – depending on what deal you select
• 3000 mailboxes (Email accounts)
• A full 97 day money back guarantee
• 24 hour technical support over email – normally I get replies within 2-4 hours at most
• Online web based control panel – although some complain about it, use it for 15 mins and you get the hang of it. I am quicker at using it than cpanel x!
• Multiple shell and ftp users – from 75 to 775 users!
• Set up new users and if you want, resell some space you are not using (I do this with a few friends – although I just don’t charge them!)
• POP and SMTP access
• You actually get one free domain regestration (for one year), with free whois protection – hiding your address
• Custom dns with all domains reg’d at dreamhost
• Support for: PHP 4, PHP 5, full unix shell, crontab, ruby on rails, ssi, cvs repository, ssl certificates (free with some packages)
• Quicktime streaming
• VPN access (with some packages)
• One click installs for things such as WordPress (includes loads of themes as well so you don’t have to download them then upload them and install them), phpBB forums, ZenCart (e-commerce), Joomla, MediaWiki (same software as Wikipedia.org) and more
• Unlimited mysql databases

And more…

Although, yes, I am hosted with them and links in this post are affiliate links to save you money I 100% reccommend them – I have a LOT of domains, LOTS of sites and use several hosts, and Dreamhost are by far the best.

You can save upto $95 by using HOST95 as your promotional code. This page has a table on it showing how much you save – often $95!

As you can see, it is well worth using it. It is no scam, no rip off. Just a great deal!

Also, if you donate money through dreamhost to charity, they match that amount (so donate $10 and the charity gets $20!).

There is a bit more information at this page about how to save money when signing up for dreamhost.