Announcement

Collapse
No announcement yet.

Robots and spiders

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Robots and spiders

    Hi

    Taken from http://www.seekport.co.uk/seekbot/ is it necessary for the following, I thought not.

    Robots

    <meta NAME="robots" CONTENT="index,follow">
    <meta NAME="robots" CONTENT="noindex,nofollow">

    The crawler reads this information to discover if the author has given permission to crawl the page.

    * (CONTENT="index") to give permission
    * or (CONTENT="noindex") to deny permission.

    #2
    Robots in text file or META format is only needed to try to stop a spider from going somewhere on your website. (I say try because several spiders ignore the robots file).For example a test folder. Follow and index is the normal behaviour of spiders.

    W3Schools have some good information on this.
    Bill
    www.egyptianwonders.co.uk
    Text directoryWorldwide Actinic(TM) shops
    BC Ness Solutions Support services, custom software
    Registered Microsoft™ Partner (ISV)
    VoIP UK: 0131 208 0605
    Located: Alexandria, EGYPT

    Comment

    Working...
    X