SEO and Spidered links

[jacy]

Hi there.
I was trying out this new SEO page i found at http://www.webconfs.com/
I was just curious as to why didn't it spider my whole site?
I notice there are a couple of cgi bin/no login type entries
could this be a problem for page rankings?
How do i add the site map to my home page?


Spidered Links :
http://www.jacyscomputers.com.au
acatalog/About_Us.html
acatalog/Contact_Info.html
acatalog/Payment_Options.html
acatalog/Privacy.html
acatalog/Terms___Conditions.html
http://www.jacyscomputers.com.au/cgi...html&NOLOGIN=1
http://www.jacyscomputers.com.au/aca...l_O_Notes.html
http://www.jacyscomputers.com.au/acatalog/Web_Cam.html
http://www.jacyscomputers.com.au/aca...Drom_CDRW.html
http://www.jacyscomputers.com.au/aca...Drom_CDRW.html
http://www.jacyscomputers.com.au/aca..._Handycam.html
http://www.jacyscomputers.com.au/aca..._Handycam.html
http://www.jacyscomputers.com.au/aca...ics_Cards.html
http://www.jacyscomputers.com.au/aca...ics_Cards.html
http://www.jacyscomputers.com.au/acatalog/Antec.html
http://www.jacyscomputers.com.au/acatalog/Antec.html
http://www.jacyscomputers.com.au
/cgi-bin/ss000001.pl?PRODREF=kt031a&NOLOGIN=1
/cgi-bin/ss000001.pl?PRODREF=hm355&NOLOGIN=1

Cheers,

Jacy

[rmladden]

A user would not be able to click on the CGI links. They go nowhere because the Actinic CGI uses the refering page to try to go back to.

Create a robots.txt file in your root folder and put this in it:
User-agent: *
Disallow: /cgi-bin

[jacy]

Thanks Bob,
I added the robots text and the site map but it isn't picking up the site map. Any ideas why?

Cheers,

Jacy

www.jacyscomputers.com.au

[rmladden]

I don't think any spider can follow the javascript tree menu. I just tried this simulator myself and all it seems to do is check the home page for obvious links (<a>). It does not go any deeper and does not seem to read robots.txt. Not really much of a simulator.

[jacy]

Thanks Bob,
I was worried i had missed adding something!!

Cheers,

Jacy