Announcement

Collapse
No announcement yet.

High bandwidth issue from access to search highlight script

Collapse
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
    #1

    High bandwidth issue from access to search highlight script

    Hi,

    We have a client who has in the last month quadrupled their bandwidth usage. Checking the stats 80% of the traffic is for the sh000001.pl file, and googlebot.com constitutes 80% of visiting sites. The site normally uses about 500mb but has surpassed 2GB already this month.

    The site is 8 months old so this latest month is an anomaly. I realise the script is used to highlight search phrases and the site has approx. 11,000 products but 1.7GB of the traffic is for this script.

    My question is has anyone else experienced this?
    Tags: None
    #2
    They are only using 2gb on an 11,000 product site! Bejesus! Does sound odd though, do you have a url we can look at?

    And I know this is a stupid question, but we are definately talking bandwidth here and not disk space?
    Cheers

    David
    Located in Edinburgh UK
    http://twitter.com/mcfinster

    Comment

      #3
      stop the search engines from following your cgi-bin in your robots.txt?

      Comment

        #4
        Originally posted by gabrielcrowe
        stop the search engines from following your cgi-bin in your robots.txt?
        Unless the entire site runs through the cgi-bin!
        Cheers

        David
        Located in Edinburgh UK
        http://twitter.com/mcfinster

        Comment

          #5
          now who would be stupid enough to allow something as daft as that huh?

          *looks around shiftily*

          Comment

            #6
            hmmmmm i wonder Gabe

            Comment

              #7
              It is a little strange as Google will normally only use the search scripts if there are links that use it or someone is doing lots of searches with the google toolbar installed and PR enabled. Google isn't in the habit of searching sites blindly.

              There are a few thing you can do to limit google searches, but on the whole, if Google wants to search a site then that is a good thing and I wouldn't want to put them off unless it was causing a problem with duplicate content or similar (and even then Google is pretty good now at dealing with this kind of thing quite efficiently).

              Bandwidth is cheap these days so my advice would be to live with the bandwidth usage, but spend some time understanding how or why it is happening to make sure there isn't an underlying problem. Nothing we can do here on that front without a url to look at.

              Mike
              -----------------------------------------

              First Tackle - Fly Fishing and Game Angling

              -----------------------------------------

              Comment

                #8
                Site URL

                Thanks for your interest guys.

                The site URL is http://autoone.co.nz/

                The 2GB figure is bandwidth. The site itself is using 39MB of disk space.

                We've got about 70 Actinic sites on the same server and this is the only one to have this issue.

                Thanks,

                Duncan Brooking
                1st Web design

                Comment

                Previous template Next
                Working...
                X