Strange Log Activity

[steveaich]

In the last month we have seen a great increase in the number of error pages diplayed, also in the number of times robots.txt is accessed (39857 times in Mar so far, cf 41000 in feb for the whole month) We also seem to have lots of hits from the ip range 80.132.xxx.xxx, these are not taking much data, but are producing lots of 'hits'.

Anyone got anyideas if the thigs are related, and if it is anything to worry about.

We seem to be getting a lot more traffic, but not more orders!

[Mike Hughes]

Obviously robots.txt is the file search engines use to make sure they don't misbehave on your site. 40,000 hits in two weeks sounds very excessive though. Are these hits all coming from one address?

Error pages are sometimes a result of search engines checking what your error page looks like. I would suspect though that it's probably a dodgy link somewhere. Can you look in your logs and see what the errored page and referring page is?

In terms of the IP address, why not post the full address so we can check who it is? You can always edit it out later once we've done some checking.

Mike

[Duncan Rounding]

We also seem to have lots of hits from the ip range 80.132.xxx.xxx, these are not taking much data, but are producing lots of 'hits'.

Maybe someones hot linking to your images.

[steveaich]

In terms of the IP address, why not post the full address so we can check who it is? You can always edit it out later once we've done some checking.

Was quoting an ip range, rather than an address. It seems we are being visited by JobSpider about once or twice a minute according to the logs, it just looks at robots.txt and nothing else, all coming from the IP range as above, I've blocked this with .htaccess, hope this is the right thing to do.

[wjcampbe]

Well 80.132 is part of Deutche Telekom - which means you are banning a sizable proportion of an entire country if you used that range.