Please urgently check my robots.txt file!

[paulbeckett]

Hi all

We have moved servers (to Host-it) and are getting some strange results in Google.

Please can you have a look in http://www.kjbeckett.com/robots.txt and confirm if this will ALLOW all robots to index all areas of the site.

I am concerned the hosts have incorrectly replaced our robots.txt file with one that prevents robots from crawling our site.

The file contains this:

User-agent: *
Disallow: /

I am worried that the Disallow: / is a blanket disallow for the whole site.

Paul

[Duncan Rounding]

That will block all robot access to your site and should be removed asap.

[paulbeckett]

Thanks. Host-it have removed the file so we have no robots.txt file at all now.

They are claiming that the file is definititely the one on our old server and they simply migrated across.

Paul

[meden]

It does seem strange that a host would be so proactive as to implement a robots file without being asked, and to make such a massive mistake with it to boot!

Did you part with your old host on good terms?

[paulbeckett]

It does seem strange that a host would be so proactive as to implement a robots file without being asked, and to make such a massive mistake with it to boot!

I have figured it out... when the new server was in test, the host took a copy our our site from one of our test servers (not the live one). The must have taken the robots.txt file from test too.

Once we had fully tested the new server, we then did a full refresh/upload and I thought the host copied over the extra files on the live server (e.g. htaccess, robots, favicon) but clearly they didnt.

Thus our test robots.txt file went into live on our live site :-(

Did you part with your old host on good terms?

Host-it actually took over Pinbrook. To be fair to Host-It, beside this problem, the experience has been fantastic... the new server is about 10x quicker but for the same cost, the support staff are very knowledable, the servers are all inhouse so they can make changes immediately. Apart from this issue I am really pleased!

[paulbeckett]

Hi all

Given the above thread.. I have a follow-on question that I would appreciate your opinions on...

We have been regularly indexed on Google for 6 years with good placements. Over the last two weeks we have been incorrectly preventing robots from crawling our site... as a consequence for some results our placement is still in Google but the listing has been rationalised to basic data.

Now that we are allowing robots to crawl our site again we hope there wont be a big impact.

Does anyone have any experience of temporarily preventing Google from crawling your site, and if so, where there any long term negative effects?

[Sean Williams]

I'd have thought it wouldn't matter much provided the website hasn't change much in the 'invisible' time.
Google isn't vindictive so it should be able to carry on as before.

[fergusw]

All other factors being the same you should see it restored to its previous ranking results.