Duplicate Content - Friendly URLs have negative impact?

[kes]

Hi all,

I've implemented freindly URLs on the site i.e.

http://www.domain.co.uk/acatalog/productcategory.html

However, the unfreindly URL is still being indexed:

http://www.domain.co.uk/cgi-bin/ss00...1234&NOLOGIN=1

Which means i now have 2 duplicate pages in Googles index.

Now i don't want to just 'turn off' or block this URL

http://www.domain.co.uk/cgi-bin/ss00...1234&NOLOGIN=1

because it is ranking and i don;t want to lose any Google love it already has.

I'm thinking block access to http://www.domain.co.uk/cgi-bin/ss00...1234&NOLOGIN=1 type URLs via the robots.txt file and/or create 301 redirects from http://www.domain.co.uk/cgi-bin/ss00...1234&NOLOGIN=1
type URLs to friendly URLs such as http://www.domain.co.uk/acatalog/productcategory.html


But is there any in built features i can use to resolve this issue?

The friendly URLs are great but if they create duplicate content then this could cancel out any benefit the friendly URLs generate.

The friendly URLs could in fact have a negative impact...

Thanks in advance,

Kes

[kes]

Hi again,

I just did a test and the redirect from the unfriendly URL to the friendly URL returns a 200 status code which tells Google that the unfriendly URL still exists and is why it remains in G's index as a duplicate page...

http://gsitecrawler.com/tools/Server-Status.aspx

Is there a way to make this a 301 permanant redirect?

Cheers,

K

[pinbrook]

assuming you have both types listed, you dont really have control over when google will drop one in favour of the other. although it has to be said the url cant be that unfriendly it is has been indexed.

if it were me i'd block access to the cgi bin for google via robots file and create htaccess for 301 redirects

google indexes quite quickly nowadays.

hopefully your friendly urls rank over the unfriendlies

[leehack]

+1 for me, block the cgi-bin.

Think long term on SEO tasks, never short term IMO.

[kes]

Cheers for the replies guys.

I think i agree the only way is to block access to the cgi bin via the robots.txt file and create 301 redirects.

Really not sure at this stage how to create the redirets dynamically though.

It seems a shame that the friendly URL feature which is supposed to have a positive impact on SEO has a side effect that has a negative impact on SEO.

Why don't the unfriendly URLs just 301 redirect to the friendly URLs by default?

1) i've got every page duplicated in G's index and G doesn;t like duplicate content. They are quite clear on this http://googlewebmastercentral.blogsp...e-content.html

2) some of the unfriendly URLs are ranking above or inplace of the friendly URLs

And to top it off the unfriendly URL redirects to

http://www.domain.co.uk/acatalog/product_name.html#a833

rather than

http://www.domain.co.uk/acatalog/product_name.html

Isn't there a built in fix? If duplicate content wasn't an SEO issue then no probs but Google is quite clear that duplicate content can cause SEO issues.

If not, any tips on dynamically generating the 301 redirects very welcome...

Ta,

K

[Mike Hughes]

If you block the cgi-bin then the redirect shouldn't be an issue.

Often the reason the cgi-bin link ranks best is because it's on the home page as a 'best seller'. Static links for these would clearly be better, I suspect it's just actinic taking the easy route when generating these links of not bothering to see if accounts are used or not.

The redirect is probably done in the scripts so it shouldn't be too hard a job to change the response to a 301.

Mike

[kes]

Hi Mike,

just blocking the cgi bin alone is not enough because any Google love the unfriendly URLs/duplicate pages have gained will be lost.

To ensure Google love is not lost a 301 permanant redirect must be put in place from the old URL (the unfriendly URL) to the new URL (the friendly URL)

Not sure how to access the scripts just yet as i've only just taken on this site.

And still confused as to why a feature that is supposed to have a positive impact on SEO creates a negative impact.

If i'm missing something or anyone can point out where i'm going wrong i would be grateful...

[pinbrook]

best sellers etc on the home page will only work with cgi-links.

if you dont use them and choose a nav system that uses friendly urls - actinic works as you want it too.

personally i've never been that bothered about cgi-links as google follows them quite happily nowadays

[jont]

I used to block the cgi-bin from Google but developments in their algorithms saw me remove the block and I gained a slight improvement in rankings.

Over time the duplicate page issues seem to have sorted themselves out - I guess Google is not that daft and realises what is going on.

I never block the cgi-bin on new sites nowadays.

[kes]

hi guys,

Google may have gotten better at handling unfriendly URLs but it still doesn't handle them as well as it does friendly URLs

And i still have both the firendly and unfriendly version listed in the index which Google doesn't like.

Best sellers isn't really i issue since these are temporary pages...

[Mike Hughes]

I've had a look at the scripts and it isn't an easy change to make.

It looks like the "200 OK" response is created by a PrintHeader routine inside Actinic.pm that is common to all the html generated by the scripts. Changing it would affect all the HTML output generated by the scripts (such as the cart, checkout pages, search results, bounce pages, etc)

To make the change work you'd need to be able to identify what was triggering the output and I'm not sure that information is available to the PrintHeader routine. You could look at the HTML being output and try to determine the correct response based on that but you'd need to be able to tell the difference between product page links and other redirects / bounces.

I can only suggest you add a request to the wish list as this would be a useful feature.

Mike

[leehack]

I'd be very surprised if the database couldn't be interrogated using php to find the html page name. I know for a fact that you can find the parent page name from a duplicate for example because I do that myself on a number of occasions.

I thought Gabe had provided a tutorial to scrap the cgi bin linking on marketing lists, pretty sure I read that in here quite some time ago.

[Mike Hughes]

OK. I've thought about this a little more.

One way of doing it would be test the HTML output for linking to the product reference. i.e. look for ".html#" in the string. That plus a conditional test for length (or ".pl") to ensure you aren't looking at a search results page should be enough to ensure that only the correct pages are given the 301 response.

You could also strip the product reference # tag from the link at this stage if you wanted.

So maybe not too hard after all.

Mike

[grantglendinnin]

Remove CGI-BIN Links - this is the thread you mention Lee, very useful stuff.

[Mike Hughes]

Looking into it, the problem seems to be the way actinic is doing the redirect in the scripts.

It looks to me as if they're creating a page with a meta refresh to the new url and then outputing that with a 200 OK response. This used to be OK with Google but from what I can tell they now want a proper status to go with any redirects.

What Actinic should probably be doing is a perl 301 redirect

$q = new CGI;
print $q->redirect(" http://www.new-url.com/ ");

or

my $URL = "http://www.askapache.com";
print "Status: 301 Moved\nLocation: $URL\n\n";

There's a lot of stuff going on in these parts of the scripts and they're used for a lot of different pages so I think you probably need Actinic to make any changes here themselves.

Mike