cgi bin and crawlers

[Doc]

HI
I recently came across this thread "Links via CGI-Bin are bad for Search Engine Positioning "
http://community.actinic.com/showthread.php?t=15747

Also this thread by domino1:
http://community.actinic.com/showthr...t=link+checker

What I am seeing (through Linkchecker) is a vast amount of links going through the cgi bin on my site. (cgi-bin/os000001.pl).

This seems to confuse bots when the crawl my sites, and sometimes I even get the cgi-bin showing up in the SERPS. Link checker also shows that apparently some of my pages should exist in the cgi-bin and they dont. This is I am assuming because of the links that passes through the cgi-bin confuses the crawler.

I think the cobination of links going through the cgi bin and the actinic sitamap causes serious crawling problems.

I also agree with the poster in the first link that this has got to be fixed by Actinic, as it is causing crawling/ranking problems as well as lose of anchor text and thus poor ranking of internal links.

As a workaround, I have stopped bots crawling the cgi-bin and have created a new site map all hard coded and have taken off the actinic generated one in /acatalog/. My site map sites in the root folder which is where it should be and links directly to index.html. That hopefully should force the crawlers to go through that route.
I know that some of you are manually creating nav links as a workaround too...that must be a pain to say the least...

[Mike Hughes]

Yup, it's a pain.

And as you say, the search engines (and google in particular) will now follow links via the cgi-bin and this leads to them seeing the same content in two locations. Duplicate content is not liked by the search engines. At a minimum one of them will be dropped, but the consequences could be worse if they figure it's an attempt to spam them and apply penalties.

Mike

[pinbrook]

I use the actinic cgi navigation as i prefer it as a method to ensure all my links to sections etc are compleltely up to date after catalog changes.

I also take the sitemap and edit it to the links only and place this at the bottom of each page (act_primary) so there are spiderable links for SEO. (I update these links less often - maybe once a month)

I also have a robot.txt to diallow spiders from the cgi bin.

Works for me

[RuralWeb]

I tend to always use a clean template for designs. This does not have navigation via the cgi-bin. If I do use the cgi-bin nav then I dont have a site map. Its one or the other I think.

Seems to work OK.

[remmo]

Is this refering to site wide links - ie either cgi links through entire site or hard coded links, or is it refering to the home page only?

[Doc]

Is this refering to site wide links - ie either cgi links through entire site or hard coded links, or is it refering to the home page only?

Internal links (section and product links) going through the cgi bin. I have 92 links going into the cgi-bin from the catalog section , and 52 going out including some .js javascript.

[jont]

Internal links (section and product links) going through the cgi bin. I have 92 links going into the cgi-bin from the catalog section , and 52 going out including some .js javascript.

Due to some serious efforts on the handcoding front only have the view cart and checkout links through to the cgi-bin

[Doc]

Due to some serious efforts on the handcoding front only have the view cart and checkout links through to the cgi-bin

How did you manage that? can you give example pls

[jont]

All the navigation is hard coded into the templates - the only links I can't remove are to the view cart and checkout as these are pointing to scripts in the cgi-bin.

As I have relatively static departments updating the main navigation is only a "slight" inconvenience

[Doc]

All the navigation is hard coded into the templates - the only links I can't remove are to the view cart and checkout as these are pointing to scripts in the cgi-bin.

As I have relatively static departments updating the main navigation is only a "slight" inconvenience

The forms are usually cgi scripts...but the rest you can change as you suggested..but I think it is better to avoid as much as possible hard coding...I have already quite a bit of that on my site..from maintanance point of view its not the way to go...better for actinic to fix the problem once and for all...I dont think it is major, I guess what is needed is a radio button to turn off cgi links. The internal mods cant be all that great. someone said in Agust that its on the wish list...is it going to be implemented? Maybe all those who are interested should email actinic...

[jont]

I think it is better to avoid as much as possible hard coding...I have already quite a bit of that on my site..from maintanance point of view its not the way to go

Your original thread was about spiders going through the cgi-bin and getting confused. From previous posts regarding your experience with SEO and getting to the top of the pile I would have thought you would have ditched the cgi-bin in lieu of direct links. Whilst it would be great if the issue was resolved in future releases (v8 or v9) I am unprpared to wait to see if resolved as customers will not sit and wait they will go elsewhere if another site is easier to find on a search.

I personally consider the hand-coding part of the whole SEO process - anything to help someone (or a spider) find what is on your page needs attention. Yes it slows things down not having Actinic take care of the navigation but IMHO that is a small price to pay - in my case a very small price due to static sections and the assumption it is all part of the SEO workload.

[Doc]

Your original thread was about spiders going through the cgi-bin and getting confused. From previous posts regarding your experience with SEO and getting to the top of the pile I would have thought you would have ditched the cgi-bin in lieu of direct links. Whilst it would be great if the issue was resolved in future releases (v8 or v9) I am unprpared to wait to see if resolved as customers will not sit and wait they will go elsewhere if another site is easier to find on a search.

The cgi-issue has only become aparent now after we were trying to figure out why the anchor text was not showing up (It affects G much more then Y or MSN)...it is a maintanace issue if you are running multiple sites to keep checking and tracking what is going on and updating your hard coded nav..and i dont really see why actinc is not fixing it ..as there seems to be strong demand from the user community for this..it makes sense...I see this as a major structural weakness in Actinic...

[RuralWeb]

The cgi-issue has only become aparent now

As you have previously stated on numerous other threads that you are an SEO "expert" I find it very strange that you did not know about the cgi issue it is very well know in SEO circles as it not only affects Actinic.

I dont believe it is a problem for any site I have built with Actinic they all rank well with Google.

What people should remember is that Actinic is a shop in a box ie a starting point for merchants who want a cost effective website. If people want a custom designed store they either do it themselves or pay a designer many thousands of pounds to produce what the client wants.

I personally think Actinic is the best on the market at the price and given the fact that with DW it can be customised very easily there is virtually nothing it cannot do - with the right skills of course! Dont blame Actinic.

[jamescheema]

Being a complete newby i'm worrying slightly about this cgi navigation problem for search engine crawlers. Would i really need to customize alot of the code to avoid poor rankings?

[wjcampbe]

No, but if your rankings are middle of the road to good, you could - all other things being equal - take the final step to the top by massaging the code.