Robots.txt file disallow acatalog??

[June]

Thanks for your responses, which has put the issue into more perspective for me.

Just one point -
Surely it is a good idea to use the robots.txt file to disallow the search engines from the CGI-BIN and LOGS areas - as these are not relevant to the search engines who are looking for information on the site - surely these areas just cause confusion, or am I barking up the wrong tree again?

[RuralWeb]

If there is nothing of interest to be spidered they will not bother. The main problem is making a mistake adding the robots text file incorrectly, you could block the whole site. If you dont need it dont use it - Actinic sites work well without messing with robot text files.

[June]

I understand the webstats had been indexed by the search engines, does this not matter either?

This is why the robots.txt file was created in the first place

[RuralWeb]

They should not be visible to the spiders if your server is structured correctly?