Bots are programmed by the people running them to use Unix and Windows commands to look for file lists - this is why, even on a Unix server, you will see a Windows folder path showing up in your logs. If not blocked by an exclusion in robots.txt even a 'good' bot will index every link it finds.
"Bad" bots will ignore the robots.txt and just index everything it can find anyway. I password protect the folders I don't want the bots to enter, as well as posting a robots.txt for the good bots.
I view a properly written robots.txt file as beneficial to the SE's telling their bot where not to waste its time. Imagine how much more often G could crawl your site content, if it was not wasting time following all the cgi links on all the sites that don't use a robots.txt file?
"Bad" bots will ignore the robots.txt and just index everything it can find anyway. I password protect the folders I don't want the bots to enter, as well as posting a robots.txt for the good bots.
I view a properly written robots.txt file as beneficial to the SE's telling their bot where not to waste its time. Imagine how much more often G could crawl your site content, if it was not wasting time following all the cgi links on all the sites that don't use a robots.txt file?
Comment