Google Sitemap errors

[AndyBorrett]

I have made some drastic SEO changes to our site that have resulted in us getting lots more visits but lots and lots of errors withing the Webmaster Tools of Goggle.

I have resubmitted the sitemap created using the Moleend Product Mash

There are 85 errors for URLs in sitemap.
The vast majority point to renamed pages and are linked from what appears to be a search result of some kind. Here is an example

http://www.seriouslysilver.co.uk/cgi-bin/ss000001.pl?PR=1&SS=bangle&SX=0&TB=A

Is there a file somewhere that I need to 'flush' to get rid of these old pagenames?

I also get 233 not found errors. These all seem to point at a webpage I deleted sometime ago from the days we were in a linkexchange thing.

http://www.seriouslysilver.co.uk/Links.php

Same question as above really - how can I clear these errors.

[leehack]

Ban the search engines from the cgi-bin using robots.txt UNLESS you are using cgi-bin navigation (a la smart theme etc.). That way Google will not spider them and your sitemap creation tool should not list them in the first place.

[AndyBorrett]

I don't use CGI for navigation apart from things like the marketing lists.

I have checked the output of the Moleend Product Mash and there is no mention of anything apart from the sections.

If I could find the file that is/was holding this out of date information I could delete it and let Actinic recreate it then setup Robots.txt to keep Google out of the bin

[meden]

What you may find is that the file no longer exists and Google is referring to a link in a page it indexed a while ago - Possible fix is to submit a removal request via Google dashboard (if you've confirmed that the source page is no longer there)

[AndyBorrett]

A couple of months after I removed the links Page I discovered that the actual link to it was just commented out and read on the forum that Google would still read this as a link so I deleted it completely.

I did however forget that Actinic seems to leave deleted pages on the server so I manually deleted all the HTML pages then refreshed -this was a couple of days ago.

I will have a go at requesting a removal of the links page from Google but don't think that will work for the CGI pages as they are not real pages AFAIK

[Dorian]

Hi Lee,

I was thinking about using a robot txt file:

User-agent: *
Allow: /

.. and I'd like to use it to stop Google coming up with some errors in it's indexing (according to Webmaters Tools, like Andy was getting).

How do I tell if I'm using cgi-bin navigation?

I'd like it to prevent Google seeing duplicate descriptions such as these:

/acatalog/The_Original_Green_Log_Maker.html

&

/cgi-bin/ss000001.pl?PRODREF=28&NOLOGIN=1

Currently it saying they are the same thing. Is using the robot txt file to stop Google indexing these the way to go?

Dorian.

[pinbrook]

How do I tell if I'm using cgi-bin navigation?

hover over your links and you will see if the left hand nav uses cgi links or not.

even if the nav is not cgi you may still encounter issues if you use bestseller lists as these use cgi.

[Dorian]

Thanks Jo, sorry to be ignorant - but I've hovered over - what then am I looking for? I currently see the alt text of the link I'm hovering over and yes, I have new products and bests ellers on the home page.

[pinbrook]

make sure your browser shows the status bar, then hover over a link - the url will then be visible (bottom left of browser - i'm using firefox)

it the site is mustbegreen site - then your left nav is not cgi - its in the format domain/acatalog/page.html. However your special offers hover over august special offers are, ie domain/cgi-bin/ss000000 etc

[Dorian]

O.K. I see that now - yes my section links are normal URL's but my New Products and Special Offers have CGI in the title.

So - if the CGI is NOT used for the main navigation, should I add it into a robot txt file to prevent Google seing it as duplicated descriptions as it currently is? If so - how do I do this?

[animal dreams]

Try googling robots.txt cgi-bin

[Dorian]

Will do - thanks for the tip Alan.
O.K. I'll add a robot with Disallow: /cgi-bin/ in it and then see if this fixes it.
Are there any down sides to doing this?

[Dorian]

O.K. - this morning I checked Webmaster Tools and my Sitemap has a great big red X against it. I can only oresume because I have added the robot file yesterday? I'll remove the Disallow: /cgi-bin/ and see if my sitemap comes back.

[cobbler]

Dorian, Have you tried testing your robots.txt file in google's webmaster tools?

(Go to Webmaster Tools | Site config. | Crawler Access | open the Test Robots.txt Tab )

This will tell you if you have a problem with it without the need to remove it and wait for your sitemap to be resubmitted.

[Dorian]

Hi, yes I did test the origonal robot and got a 200 code back, which I believe means all's o.k. However, I hadn't tried the CGI one - so I will do that now.