stats error

[shabbie]

i was checking the stats for www.siam925.com today and noticed (not for the first time) that the most frequent download error is for /robots.txt
what is it and are there any recomendations to fix it?
other errors are for pages that i deleted from previous site set up (pre-actinic) i figured that these page errors would iron out as the search engines find our new pages.
please keep your replies simple coz im no graduate!

[jont]

Sharon - do you have a robots.txt in place on the server? This would be created by you and not part of the Actinic install.

Some search engines will look for the robots.txt file ... it is a small text file that essentially tells search engine spiders where they can and can't go

[Duncan Rounding]

If you don't need or wish to stop the SEs from crawling certain pages on your site then you can just delete the robots.txt file as it is not needed.

The other errors should eventually disappear but it might be a good idea to point that traffic to your new site by adding redirects in an .htaccess file.

[shabbie]

If you don't need or wish to stop the SEs from crawling certain pages on your site then you can just delete the robots.txt file as it is not needed.

The other errors should eventually disappear but it might be a good idea to point that traffic to your new site by adding redirects in an .htaccess file.

i dont have any issues with the SEs crawling any pages, although i do have some items ticked as do not show on website on actinic.
cant seem to find robots.txt in ftp files, is that where they are stashed?
also where would .htaccess be placed?
cheers

[jont]

Both should be in the root of the server folder ie the folder beneath acatalog/

[Duncan Rounding]

It could be that some other crawler is looking for the robots.txt file but can't find it. In your site logs you should see the referring page or domain that might help you determine who's looking for it.
I wouldn't worry about it.

[shabbie]

ive recently registered my site will loads of the freebie SEs so hopefully it will resolve itself then? i was just worried that i might be losing customers.

[pinbrook]

When spiders visit your site, they check for a robots.txt file to see if you have any instructions for them as to where to go or where not to go.

Having a robots file is purely a choice thing. If you are happy for spiders to roam everywhere you don't have to have one. I use a robots file, for example, to keep spiders out of my cgi-bin and image folders.

Your original question was asking why you had error messages in your stats which were being caused by spiders looking for a robots file that wasn't there.

Thus to prevent this error showing in your stats as an error you should upload a robots file.

If you decide you don't want a robots file as you are happy for spiders to roam freely, you don't need one but you will continue to get this "error" message in your stats.

[shabbie]

I use a robots file, for example, to keep spiders out of my cgi-bin and image folders.

would i set this up within htdocs then? what would a suggested file look like?

[NormanRouxel]

Just google for "robots.txt tutorial" and you'll find lots of help.