robot.txt

[Olly]

I have been reading up on robot.txt files for the last couple of days now!

Am i right in thinking that if you do not wish to disallow any bots or directories then there is no need to have a robot.txt file?

Also are there any particular files that should (or would be a good idea) to disallow? e.g. cgi-bin or shopping cart etc.

Thanks

Olly

[Mike Hughes]

Yes. Unless you specifically want to prohibit a search engine spider from going into certain areas then there is no need for a robots.txt file.

I don't worry about them going into the cgi-bin as they aren't going to cause any problems. All they do is sniff around and see if anything looks interesting.

Mike

[jont]

Best not to exclude the cgi-bin - I excluded them as a test on a site and there was a noted decrease in listing positions which was directly attributed to the robots.txt file exclusions - as Mike says they will just bounce around in there until they get bored but to no adverse effect.

[Pneumatus]

If you don't need a robots.txt file, the easiest thing to do is to make one with:

User-agent: *
Disallow:

That just means that all user agents can go wherever they want, plus it saves on error log files when spiders request it, and most logfile stats generators use hits on robots.txt to give you stats on spiders anyway (like awstats).