Do I need a Robots.txt

[robmob]

I have been trying to get to grips with all the stats my hosting company provide and have noted that one of the largest page failures is for the request for robots.txt.

Can anyone advise if I really should have this file and if so how do I create it and how do I get it onto the site.

Many thanks

Rob
Performance Motorcare

[wjcampbe]

Robots.txt is a file where you tell search engine spiders what they are NOT allowed to look at on your site. If you have nothing you want to hide from the search engines AND no individual spider you wish to keep away then you don't need a robots.txt.

If you do need one, you can use ftp to upload it to the root of your website.

There is more detail available here http://www.robotstxt.org/wc/robots.html

[pinbrook]

This is my robots file - with just the bits relevant to catalog. I would upload a robots file just to get rid of all the error messages in your logs.

User-agent: *
Disallow: /cgi-bin/
Disallow: /acatalog/*.cat
Disallow: /acatalog/*.fil
Disallow: /*.gif$
Disallow: /*.jpg$

this tells ALL spiders not to go into the cgi-bin, or index any cat and fil files (I'm not sure why i did the cats and fils but there must have been a reason at the time!)

I also disallow all images from being indexed to, so they don't end up in Google images and get ripped off