Question about Error 404

[runningwolf]

Quick question for you guys
I have been looking through the awstats info for my site and in the error codes bit against error 404 I get
/www.theorderofinitiative.co.uk/robots.txt 1154 hits
now the next one on list has 55 hits so does this look like a problem I need to fix?

Cheers
Paul
www.theorderofinitiative.co.uk

[NormanRouxel]

robots.txt is an optional text file that Search Engines look for whenever they enter a site. It contains a few simple instructions intended only for search engines (like don't index temp files).

You don't need this file, and these 1154 hits aren't anything sinister; just Google, etc looking for it.

Just look up robots.txt on the web for a tutorial.

[runningwolf]

Thanks Norman, so there is no advantage to having this file present on the site?
I had been worried that maybe it not being found was hurting my site being checked by seach engines.

[Mike Hughes]

No, you're fine without one. The default behaviour for all search engines is to search and index all pages unless specifically told not to by the robots.txt file.

so you only need a robots.txt file if you want to prevent a search engine from visiting certain areas of your site.

Mike