cgi bin and crawlers

[RuralWeb]

download Link Checker and look at the Site Mapper

I have been playing with this all day looking at a site created with only cgi nav and a site created with a clean template. Unless I am using the tool wrong, I can find no difference between the two sites. I have also checked a non Actinic site and it looks the same as well.

Both sites are fully indexed by google and msn with no cgi references other than the usual checkout etc.

I have also noticed that many Actinic sites that rank well use cgi navigation and it does not seem to affect them. If it were a major problem it would affect every site that used cgi nav and would be well publicised.

Also worth considering is that the text used in the anchor text passes any page rank increase to the landing page, googlebot can follow cgi links but can it pass any page rank increase to the landing page? if it cannot then this is another reason not to use cgi nav.

Actinc has many templates with only a few using cgi nav if anyone is worried they should use a non cgi template, there has always been problems with cgi nav and I think there always will be. The simple answer is dont use them!

I also dont think there is a problem - people have a choice about which type of navigation to use in the same way they can pick the type of computer they buy, PC or Mac they both have pros and cons.

[Doc]

I have been playing with this all day looking at a site created with only cgi nav and a site created with a clean template. Unless I am using the tool wrong, I can find no difference between the two sites. I have also checked a non Actinic site and it looks the same as well.

Both sites are fully indexed by google and msn with no cgi references other than the usual checkout etc.

.

Could you tell me which site you are refering to. I have checked your smoked foods site and as I informed you before there are links through the cgi bin. If you are using Link Checker, you have to use the Mapper function and then you need to look at the incomming and outgoing links at each node...
Before advocating a solution , it has to be tested and to be seen to be working..

[RuralWeb]

Im not advocating a solution as I dont beleive that there is a problem. I am using Link checker pro and the mapper as you suggested.

Yes the smoked food website has some links through the cgi bin BUT they are of no importance as they control the shopping cart. It uses a clean template so all important links are direct. Every page is indexed and it is No1 for its major keywords.

The site using cgi links is also fully indexed and is begining to get good rankings although it is still in the construction phase. I will not post the url as I believe that as you seem to be the only one experiencing/understanding the problem you should post yours so that it can be seen.

Anyone else got a view - I getting bored so I will get back to site building.

[Duncan Rounding]

Without a specific URL (and specific links in Link Checker/Mapper) that show the problem how can anyone judge the issue.

[Doc]

Links through CGI BIN

Here is a Picture of the Inbound Links flowing from the SITEMAP to the CGI BIN....Huge Problem

[Duncan Rounding]

Interesting. I've run my site, built from a clean theme, through Link Checker/Mapper and don't see this problem. I have all my pages linked through the side navigation and sitemap (with Actinic generated hard code links).

The way I've built my site is such that it has sections with overviews and child sections each with only one product as a detail. The link from the overview page is via cgi-bin to the detail (although both the detail and overview pages are also in the Actnic sitemap as hard links).

I have no 'red' bad cgi-bin links except a few that have timed out either due to the timeout settings in Link Checker or my slow connection here. The only other 'bad' links I have are those where I have js to generate a pop-up for additional images of a product.

I presume there must be other products around that do the same as Link Checker - might be worth comparing another utility to double check the results you have for your site.

Duncan R

[RuralWeb]

It would have been nice to see this earlier. I have repeated my scans on a clean theme site, smart theme (no sitemap) and a Dreamweaver non Actinic site and they all do not have the problem you are showing.

They do have some cgi links but as I have said they are only for the shopping cart etc ALL important links are direct.

I agree you do have a very strange looking site scan, but I think any problem is confined to your site. I would also do a double check with another software package, if it shows the same results I would seriously consider a rebuild.

[Doc]

Interesting. I've run my site, built from a clean theme, through Link Checker/Mapper and don't see this problem. I have all my pages linked through the side navigation and sitemap (with Actinic generated hard code links).

Duncan R

Duncan
These are some of the links from the cgi-bin in your site:http://www.fancylines.co.uk/

[Duncan Rounding]

I need to understand this better.

When I run Link Checker/Mapper and then view the data in grid format in Mapper (talking from memory now - my laptop's off) then are these links, that you've shown red in the attachments, those noted as 'bad' in the list? If this is the case then there is certainly a difference between what you've captured and what I've captured.

I don't know LC/Mapper well (new program to me) - can you please run through the steps you took to capture that data so I may repeat it here in my copy.

Duncan R

[Doc]

I need to understand this better.

When I run Link Checker/Mapper and then view the data in grid format in Mapper (talking from memory now - my laptop's off) then are these links, that you've shown red in the attachments, those noted as 'bad' in the list? If this is the case then there is certainly a difference between what you've captured and what I've captured.

I don't know LC/Mapper well (new program to me) - can you please run through the steps you took to capture that data so I may repeat it here in my copy.

Duncan R

Duncan
First you need to use Link Checker. Type in your url at the top rhbox, and it will take quite a while for your site to load with all the links, it too 1/2 hr here, maybe quicker with you. When Link checker finishes it will give you a pop-up summary window. Also at that point, no more data will be flashing at the lower lh corner. You will notice you have many links (from the cgi-bin) with Status "Missing"(about 100). These links are going through the cgi-bin.
Click on the Tree icon on linkchecker when it has stoped and completed everything (after summary pop up).
That will create a map and will automatically open up the Link Mapper with your map.
In Link Mapper when it opens and you see your network diagram, click on the 5th icon on the top row (3 black arrows going out of a red box). Then to see what links are going out of any of your cgi-bin urls, just click on any of them and you will see the links fanning out. These are the images which I captured.

[Duncan Rounding]

I have looked again at LC/M and attached a capture. You will see their are two clusters of links - one the main hard links from index and the other cgi-bin. On many of my pages I have added a link to run a search that displays Plus sizes - the result of this search is of course additional cgi-bin links, on many other pages I have standard cgi-bin links to detail product pages. The majority of links that LC/M show as 'bad request' are those that link to detailed product pages eg:
http://www.fancylines.co.uk/cgi-bin/...3489&NOLOGIN=1
which actually resolves to a page called:
http://www.fancylines.co.uk/acatalog...s.html#aDG3489
- I don't know why LC/M indicates a 'bad request' here because you can browse with these links. This is leading back to an earlier thread where someone questioned the accuracy of LC/M and whether it scans in same way as the SEs and/or your browser.
These links are already mapped via the sitemap and the only problem could be that they might not (and perhaps shouldn't) count towards internal page ranking.

In summary I don't really think it's a problem on my site.

Duncan R

[Duncan Rounding]

Just to add out of interest: HTML Link Validator from Lithopssoft has no bad responses ( as opposed to Link Checker) when reading: http://www.fancylines.co.uk/cgi-bin/...3489&NOLOGIN=1

[Doc]

Just to add out of interest: HTML Link Validator from Lithopssoft has no bad responses ( as opposed to Link Checker) when reading: http://www.fancylines.co.uk/cgi-bin/...3489&NOLOGIN=1

Linkchecker is a website analysis tool. Linkvalidator from what I see on their ws checks for broken links only. The issue is not that the links are broken, but the Links are Virtual Links as they appear to go through the cgi-bin. That is why Linkchecker flags them as "missing". you are probably getting a closer picture to how the crawlers see your site. if you check your site links on google, you will see few occurences of .../cgi-bin/....

[Duncan Rounding]

I agree you do rarely see cgi-bin links in Google. I did have a few for a while bu tthey didn't last long. Interestingly though I tried a couple of almost totally cgi navigated Actinic sites that I found searching around and ran them through LC - they too showed the cgi links as bad - but looking at the site in Google they are well indexed with the resolved non cgi names.

It still seems there may something fishy about your and maybe some other particular sites cgi navigation (or Google) - but it doesn't seem to be the norm.

[wjcampbe]

Hows about this

If the only link G find to a page goes through cgi - they list it. If they find the same content with a direct link, they drop the cgi link.

Make sense to anyone?

Seems to me this whole thread is much ado about nothing.