cgi bin and crawlers

[pinbrook]

To the best of my knowledge we are stuck with the cgi navigation as business requires it to control where to direct logged in and non logged in visitors.

Thus the choices are

ditch cgi nav in favour of hard coded links - this means more careful attention is required whilst adding sections etc

or

leave cgi nav in place and use the sitemap to also generate SEO links - easier for maintenance, but maybe a diadvantage as rural web says too many links may be problematical.

BUT the only option NOT to use is cgi navigation only.

Jont uses hard code
I use cgi nav and hard coded links generated from the site map
Rural web uses either one or the other

We have all reported successful SERPS

[Doc]

As you have previously stated on numerous other threads that you are an SEO "expert" I find it very strange that you did not know about the cgi issue it is very well know in SEO circles as it not only affects Actinic.

.

I think it only affects actinic as far as I know, I dont know what circles you are refering to.and you are incorrect when you say that it is well known that cgi-bin affect crawling. I did a search in google and nothing came up.... it seems that you are upset about something, because you keep posting these posts about me being an expert..I am only an expert to my self and those I help and my pear group ...not to you ...
I think in this forum only olderscot has raised this issue before (in august). . and I take my hat for him, becuase he rightly believe and I do that this is something that has to be sorted out..It is a small matter that can be sorted out relatively easily.And yes Actinic is a good product, and it will be even better if this issue is sorted out..at the moment it is causing problems with crawlers, especially google...
Yes I did check your website smoked food..and yes...there are many links that go through the cgi bin...So obiviously your solution did not work..
If you read carefully my post, I proposed a solution ...kinldy stop these posts of We and I agree with ...be creative and have some original thoughts...

[RuralWeb]

Below is one quote from an SEO site which talks about spidering problems. The problem as you see exists with most ecommerce solutions eg mambo which is a dynamic solution with the added problem of non static pages.

I dont know all the answers and other forum members have different solutions to the problem - if it is a problem, I dont think it is that bad. I have not seen an Actinic site that exhibits it. Yes some of my pages have cgi links but these are to things like the shopping cart which you will never be able to get rid of and it does not matter because I dont really want google to rank them! All pages that rank are urls without any cgi reference. This has happened to sites I have handcoded the navigation or used the cgi navigation (without a site map) so I dont think that users with the cgi navigation need worry too much.

If they think it is a problem then change template or it may be possible to strip the navigation element out of a clean template and insert it into a current site. Ive never tried it so I dont know if it would work. The only template I have used with a cgi nav is the Smart theme and it is indexed by google ok.

If anyone knows of a site which shows the problem I would be very interested to see it.

CGI-BIN Directories

CGI-BIN directories have been ignored for the most part by more search engines than I would care to count. The reason for this is because the URL created by the CGI scripting language includes the question mark (?) and ampersand (&).

Until a couple of years ago no search engine would even dare approach a CGI-BIN directory. Google™ can index those pages, but their success is not always 100 percent. Other search engines do not have the capability to index within the CGI-BIN directory which limits the site owner's ability to be found.

This single fact led many e-commerce site owners to find search engine optimization companies to optimize their web sites and build doorway pages linking to the items within the CGI-BIN directory.

There are two types of set up for the CGI-BIN e-commerce system.

Situation #1:

The e-commerce system is dynamic and uses a flat-file or database that presents the items and categories in a dynamic fashion. This requires the site owner to build pages external to the CGI-BIN directory that links to the items in the e-commerce system.

Situation #2:

The second situation calls for external pages that connect directly to the cart page which shows the items ready for purchase. This is distinctly different from the above situation because the system relies upon external pages that have "add to cart" buttons or links. When the link or button is clicked the link informs the shopping system that an item has been selected to add to the cart.

This situation requires the site owner to add the items to a flat-file or database for the shopping system to know the price. It also requires the site owner to maintain the external pages with the correct information. If either side of the equation was ignored the system would fail and items could not be added to the cart for purchase.

[Doc]

Thanks for the reference...but it really does not describe the problem as we have it in actinic..believe me it is not a common problem within other ecom packages.....as I said a google search draws a blank..

there are essentially two problems here:

1. Loss of Anchor Text for INTERNAL LINKS..The Anchor Text does not show up in Actinic Sites..that is quite important for ranking

2. CGI issue...
As Pinbrook has commented there are few workarounds..
My workaround is to block the cgi bin and to create a site map with hard coded links and move it into the root directory

But I hope that Actinic sorts these problems out ..because it is marketed as a Search Engine friendly system...and really it is very straightforward to sort out ...I guess if the user community wants it as a priority..maybe actinic will listen

[pinbrook]

my pear group

??????????

[RuralWeb]

Doc if you want help then post your URL so I can see what you are talking about.

[Duncan Rounding]

I'm confused Doc - what do you mean the anchor text doesn't show up. Maybe I'm misunderstanding - do you have gaps in your text where the anchor text should be or what?

Duncan R

[David@MPL]

I'm a little confused by this whole issue. On our site, links are direct unless you're logged in, but on this site (which was mentioned in another cgi related thread) the links are all via the cgi-bin and they don't appear to have a login feature. Are we saved from this problem because we're using a "clean" theme?

[Doc]

I'm confused Doc - what do you mean the anchor text doesn't show up. Maybe I'm misunderstanding - do you have gaps in your text where the anchor text should be or what?

Duncan R

Duncan
I use a tool which shows me all the incomming links (Backlins) and the internal links as well as their respective anchor text.
I used this tool to like at my site, as well as fuzzy bear's site (pet-bliss.com) and few other actinic sites. The internal links dont show the anchor text that one would expect. For an example if you have a section Name called:
Mobile Phones
and a subsection Name called Camera Mobile Phones
The Links would be:
Mobile Phones--->Camera Mobile Phones
Both "Mobile Phones" and "Camera Mobile Phones" are Anchor Text in the internal link structure.
This does not show up at all..the Anchor Text is either the Title of the Home Page . The site www.pet-bliss.com ( fuzzi bear ). Each line gives the name of the search engine, the page html, IP address , Page title and the anchor Text The Anchor Text is all "Pet Products"...which is not what should be there...In other actinic sites I have looked at, there is no Anchor Text at all...now frankly we dont know what the reason is...it could possibly be the crawling through the cgi bin which is confusing the crawler, but that is just an assumption. If the Anchor Text which is there (section/sub section name/link name) is used, that would help tremendously with the ranking

[Duncan Rounding]

Why not try running that tool on a site that's based on a 'clean' theme that doesn't use cgi-bin linking and see what difference it makes.

What tool are you using?

Duncan R

[Doc]

I'm a little confused by this whole issue. On our site, links are direct unless you're logged in, but on this site (which was mentioned in another cgi related thread) the links are all via the cgi-bin and they don't appear to have a login feature. Are we saved from this problem because we're using a "clean" theme?

I have not used a "clean theme" so I cant advice you on this. I have used mainly "Smart Theme".
But a simple way of finding out how your internal links are generated (from sitemap or cgi bin or both) is to download Link Checker and look at the Site Mapper. It gives you a very good 2D visual of all the in and out links from any point in the network. You can download a demo version.
I think the best that can be done to solve this problem is:

1. Actinic fixes it ( Hopefully Actinic will do somethig about it..we all love Actinic and have invested much in it It is a great product. . hope you guys are listening... )

2. Hard Coding the nav buttons. This is a half solution still does not solve the cgi generated forms which still allows many links to pass through the cgi bin from the forms. I think you would have to re-write some of the perl scripts to make this completely clean solution. Not practical for some users.

3. Using the Clean theme...some suggestions there..Does this really solve the problem? Some experimentation is required and checking with something like Link Checker. What about script generated forms?

4. Block the cgi bin from the bots (robots.txt). I am experimenting with that. I have no final proof that this will work. I have to see how the bots behave. Essentially you are forcing the crawlers to crawl from the site map...and disallowing them access to the cgi bin.

[gary in jersey]

hello

I also have a robot.txt to diallow spiders from the cgi bin.

what would you put in the robots.txt do disallow them from crawling the cgi bin, ?

[pinbrook]

use notepad

paste in the following

User-agent: *
Disallow: /cgi-bin/

save the txt file called robots.txt

upload to the root of your site

[Doc]

letter to actinic?

How many here would be interested in signing off a letter (email) to Actinic asking them politely and in a friendly way if they plan to fix the "cgi bin" link problem. I think this would help. Us doing nothing about it is not going to fix the problem. But if we all unite on this issue, it may well get heard and fixed. If your aim is to rank highly in your search term then fixing this problem will go a long way towards helping your site getting there.

[Duncan Rounding]

Using the clean theme, as I do, I don't think I have the issue. The only cgi-bin links I have are to the cart, contact page and additional links through images - I'm not concerned about those - all my pages are linked in more than one way with hard links generated by Actinic in the section lists and the sitemap that I hard code a link to.

Perhaps the problem is that the cgi-bin response time is slower than hard-coded links - this would then be a possible hosting issue.

Have you spoken directly with support - that should surely be the first action to take if you are so concerned - raise a support note with them.