cgi bin and crawlers

[BPJSURF]

Site navagation vs search engine crawling

All about nothing? possiably.
If your using cgi-navigation through javascripts and you do not have a sitemap then your not going to get the crawl from some seach crawlers as you may want. to over come the issue you could implement text links to core pages in the home page. This is an excellent solutions. Specifically code a text link to teh Sitemap as Pinbrook suggests and you have settled the issue for most crawlers.

There are two issues at hand. The first is useability of the web pages. The second is the indexing of the pages by search crawlers. A good ecommerce professional can easily address both by building proper links where needed and supplying a proper sitemap for indexing by search crawlers. Be careful as the Actinic site map omits all pages except the catalog pages, so you will need to hand code the informational pages into the proper template to get full coverage in the standard Actinic Site map.

There is more.......
These day the crawlers and directories are becoming more competitive in finding pages. Remember the indexes get rated on the volume of pages they index.

Google released Google sitemap for developers to address this specific issue. http://www.google.com/webmasters/sitemaps/login

This technology will build a complete sitemap into a single file for direct submission to google, ensuring you get a fully index site from Google. Yahoo offers a similar service which allows for a file submission, its less know about the inner workings of the Yahoo submission service however I have contacts within yahoo! and I have some sites that get indexed through their specific yahoo submission service.

We actually have a tool kit that runs on our linux servers to build the submission files in a specific format for both Google and Yahoo and the files are submitted for index weekly. It all runs from a cron job.

So continue to use cgi navigation elements with reckless abandon but cover your back with a google | yahoo submit and a proper sitemap. You will get indexed if you take time to make sure you cover all the bases.

hey its just an opnion, what do I know?

Brian Johnson

[wjcampbe]

hey its just an opnion, what do I know?

As much as me, possibly more.

This entire thread just churns in my stomache. Ditch the cgi-bin or suffer SE death. We must make Actinic see the error of their ways - force them to ditch the cgi scripts which provide 90% of the functionality of the software.

Come on, please. There is a perceived but unproven hypothesis that cgi-bin links are anethema to the professionals who write SE crawlers, a perfectly valid workaround has been put forward (link to site map from home page), and then we still have a diatribe like this.

I don't see this thread as helpfult to other user's (How is a new user, with no experience in SE going to react to this thread?), or to Actinic.

Brian - I agree with everything you say, for a change

[Doc]

As much as me, possibly more.

This entire thread just churns in my stomache. Ditch the cgi-bin or suffer SE death. We must make Actinic see the error of their ways - force them to ditch the cgi scripts which provide 90% of the functionality of the software.

Come on, please. There is a perceived but unproven hypothesis that cgi-bin links are anethema to the professionals who write SE crawlers, a perfectly valid workaround has been put forward (link to site map from home page), and then we still have a diatribe like this.

Here Here.. i second that.. I am not sure what type of an effort is that for Actinic, they would have to change quite a bit of the behaviour of the actinic front end, I dont think its just a small mod, but probably a major re-design...so my guess is that we are stuck with it for a while...