robots.txt

[Raj Bagga]

Can anyone please confirm where:

a. the robot.txt needs to be on my server

b. if the what it contains (as below) is correct to allow all my site to be indexed?

User-agent: *
disallow:/

c. if there are any SEO / Robot issues in relation to redirecting traffic directly to your shop by placing the .htaccess in the httpdocs folder?

regards

Raj

[Duncan Rounding]

The answers to the best of my knowledge:

a.
You do not need a robots text file on your server to be indexed if you do not need to restrict the robots from some parts of your site.

If you do use one it will need to go in the root directory of your domain or at least where a redirect sends it on landing at the domain root. If you have redirection and not sure where to put it then put a copy in both places.

b.
The content you show will disallow a full index. (edited to correct)

c. SEO should be no problem if it's a 'permanent' redirect.

[RuralWeb]

the robot.txt needs to be on my server

If you dont need to restrict acess to parts of your website then don use one.

[Raj Bagga]

Thx to both, but out if interest why & when would you need to restrict access to parts?

[jont]

the robots.txt file is often a good way to see how many "unknown" search engines have crawled the site as this is often the first thing they go for

[Raj Bagga]

As I am new to actinic & web, if possible can you kindly explain what you say in a little further detail OR direct me to some reading? PLEASE?

[RuralWeb]

This area has been discussed in depth on the forum so a search will help or you can look at http://www.robotstxt.org/wc/robots.html

[Raj Bagga]

Thank you very much I found that I am indeed excluding all robot from my site http://www.robotstxt.org/wc/exclusion-admin.html, as, as per my orginal question I stated that the contents of my file was :

User-agent: *
disallow:/

Can anyone please confirm, directly! If that is the case???
regards

[Duncan Rounding]

Oops, yes you are correct adding the '/' after disallow does stop all indexing. Removing it will allow all indexing.

[pinbrook]

I use a robots file to disallow acces to

cgi-bin
stats folders
image folders (i don'y want my images in google images)
test folders and so on

its a personal decision whether to restrict access or not

[Raj Bagga]

Thank you so much!

[Raj Bagga]

Jo,
Thx I'll take a look at my CGI bin to see if I've got any stuff I do not want the robots to index etc.... thx

[RuralWeb]

Robots files are dangerous things to be playing with you can very easily block the SEs from your site - I never use them.

[Raj Bagga]

Malcolm,

Thx but I found it on my server root directory, when just looking around, I never put it there, nor obviously know how to play about with one ?

Quite an upset! But thank you all for your help and patience's with my questions.....
regards
Raj

[Duncan Rounding]

I use a robots file to disallow acces to

cgi-bin
stats folders
image folders (i don'y want my images in google images)
test folders and so on

its a personal decision whether to restrict access or not

Interesting this. I thought that the robots followed links from the root. This being the case how can the robot get into stats and test folders etc?