Announcement

Collapse
No announcement yet.

The Googlebot has been and I'm confused!

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    The Googlebot has been and I'm confused!

    Hello,
    I re-launched my website www.turbobits.co.uk last Sunday and have been visited twice by the googlebot since. The first time it included a dozen of my pages and now its decided that only 2 pages are fit for the search engine, my site has hundreds of pages! These are the two that the 'bot liked:

    www.turbobits.co.uk/cgi-bin/ss000001. pl?SECTIONID=index.html&NOLOGIN=1
    www.turbobits.co.uk/cgi-bin/ ss000001.pl?PRODREF=TBGtech&NOLOGIN=1

    My site structure is:
    /index.html (in the root)
    /acatalog/ contains another index.html
    /cgibin/

    On my Actinic 'tree' where I structure the website I see that I have a page called 'home' right at the treetop which is the index.html in my root and then a page called 'Online Catalog' which is the index.html in the acatalog directory. Home has a few 'fragment' pages that make up the homepage of my site.

    When I look at my site online and I look at the link locations when I hover my mouse pointer over the left hand menu I see that all the lines in the menu point to this ss000001.pl script in the cgi-bin. All the link locations I see when I hover my mouse pointer over an image in a section of my online catalog have 'real' links to the html page titles of the sections and subsections.

    So.. When the 'bots come do they follow the links in the menu on the homepage and get lost in the ss000001.pl script in the cgi-bin or do they eventually manage to find their way out and into the acatalog html pages and follow the real links?

    I'm wondering if I've fallen into a common trap and made mistakes in the way I've set things up - please could you learned people please help and enlighten me? Also, do you think my workaround will work? - I've written it below in the next paragraph...

    As a workaround I've put a couple of links on the bottom of my homepage that link to the /acatalog/index.html and /acatalog/sitemap.html files in the hope that the 'bots will follow these and find real links at the locations. I've found that these two files contain 'real' links that don't go via perl scripts. (These two links are the two full stops above the first 'shopping directory' logo at the foot of my homepage if you want to see where they are!)

    Thanks

    Richard J Bridge
    http://www.turbobits.co.uk

    #2
    The dynamic lookup links on the left aren't easy for a bot to follow, even one like Google that can index dynamic pages because once it navigates through that ss000001.pl? bit it encounters a <meta refresh> tag which stops it.

    I don't know what the solution is because if you have to keep track of logged in shoppers, then you need to use the ss000 scripts.

    I think the standard advice is to include a link to the Actinic sitemap from the homepage and provide several static links to important pages within your shop such as acatalog/index.html and top level pages. This at least gives the bot something to follow so you are on the right track there. You might want to put those links as close to the <body> tag as posisble as many bots only index up to about 100kb of a page.

    Having said that, Google usually won't index all or even many of your pages first time round - it has to come back and so you should give it time.
    http://www.johnsons-seeds.com - Actinic plugins, remote add to cart and custom CMS
    http://www.dtbrownseeds.co.uk - More seeds and plants....
    http://www.mr-fothergills.co.uk - Well it used to be Actinic...

    Comment


      #3
      davet alluded to the problem - I'd say it even stronger - you shot yourself in the foot using dyamic navigation - even if google follows these links, your chances of getting results from them are slim - no other engine/bot will even attempt to follow them.

      Honestly, Actinic needs to remove this dynamic element to the navigation, it does more harm than good.
      Web Design & Ecommerce - Affordable Web Hosting
      Free and low cost Merchant Accounts coming soon..
      NOD32 Antivirus - Reciprocal Links for Actinic Sites ONLY

      Comment


        #4
        Thanks DaveT and webyourbusiness... I thought as much. This is my 4th re-launch in a year of my website. MkI was written in Word, as was MKII, MKIII was in QDCAT and used frames and now I've decided to buy 'proper' software aka Actinic. I wrote a lot of html into my QDCAT store as the search engines only ever got as far as 'This catalog uses frames'. I wrote in hundreds of links to the individual webpages from the homepage and disguised them all as grey dots on a grey background. Each dot was actually a 1x1 pixel jpg image of... a dot. They had alt text behind them too and this worked.

        I've also noticed that there is no provision for alt text in the template I'm using in Actinic and so far I've used the 'browse for images' function to insert them. I'm contemplating dumping this and coding the image links straight into each page in the near future.

        Do you think I aught to do this and also stick a few hundred "white dots" on the top line of my homepage for the 'bots to follow?!?

        Regards

        Richard
        http://www.turbobits.co.uk

        Comment


          #5
          White dots and html links seem to work!

          I had a busy day 'coding' yesterday. Was at it from 7.30am to 8pm experimenting , reading this Actinic forum, 'how to html' webpages etc and then actually writing html into my site. Anyway, here's what I found:
          *******
          The homepage in my root directory when spidered gave two results pointing to the ss000001.pl script in the cgi-bin. Like you said Actinic V7 uses dynamic linking for the menu tree. Perhaps if folk didn't have the hompage but rather just had an online catalog with the products on page1 this problem wouldn't occur as the html links from the product pictures would take the 'bots through the site.
          Anyway, my workaround was to create html <a href.. type links with alt text behind them tagged onto 2x2pixel images of a white dot and place them at the top of my text (where the 'bot will see them first!). Take a look at my homepage now and hover your mouse under the title "Our new look online shop is open for business!" until you see the links flash up in your browser lower bar.
          Next thing I found was that the 'MORE' buttons that a customer can push to get extended information pages are written in JavaScript. I came across a spider-simulator called poodle that sims the googlebot and when it came across the javascript links it did not follow them. Reading on the forum it seems that no spiders follow the more buttons! My guess is that Acinic have done this on purpose as if you type the name of an extended pop-up into your browser it comes up full screen, on its own and without the catalog. I suppose most webmasters don't put much info in this page.. just like a bigger picture and 'it comes in blue, red or purple'. I, on the other hand, don't write much on the product page but put loads of text in the pop-up!
          My fix for this also addresses something else I don't like much about the MORE button. I feel it needs something like "For extra info click More.." or words to that effect, I know you can have text buttons but here's my solution: What I have done is write "for detailed product information click..." in dark grey italic just above the MORE button. HE HE HE... but it's a hyperlink to the pop-up page in straight html. Clicking the button brings the pop-up to view in a small window with its JAVA and doesn't get the pop-up spidered. Clicking my text brings the pop-up to view full screen (coz you cant make it smaller without using java) and is spider friendly. I checked with a text only browser and a couple of spider simulators.
          This leads us onto the problem that these pop-up extended pages don't bring the catalog up. Now, if google spiders my pop-ups and someone finds what they want in a search and gets the page without the catalog what do I do? So, to fix this I have written "if you arrived here from a search engine and cannot see our whole catalog, please click this sentence to be taken to the section of our Turbobits on-line store where this product is listed" in red italic. Clicking the text brings up a new browser page with the section or subsection of my catalog where the product they were looking at is listed.
          Anyway, thats what I've done... to get to the point Google had only 2 listings for Turbobits yesterday and today its got 36!!!!! And which 36? All the ones behind my white dots on the homepage
          I thought I'd write all this here in case anyone else is interested. I'm new here and new to Actinic but feel I should share stuff like this with you all - do you mind?
          Enjoy your Sunday.
          Regards
          Richard
          http://www.turbobits.co.uk

          Comment


            #6
            Did you know you don't have to keep the popup pages as pop ups?

            That way you can set your detailed pages to indexable by search engines. If you want to see that effect:
            http://www.johnsons-seeds.com/acatal..._seeds_13.html

            This is achieved by adding NETQUOTEVAR:INFOTEXT to the standard product template - then also UNtick 'Generate Popup Page'. And you show the search engines what you want them to see, not error messages.

            When I clicked your 'Close Window' button, it does not close the window but takes you to the normal pages.
            http://www.johnsons-seeds.com - Actinic plugins, remote add to cart and custom CMS
            http://www.dtbrownseeds.co.uk - More seeds and plants....
            http://www.mr-fothergills.co.uk - Well it used to be Actinic...

            Comment


              #7
              Hello Dave,
              Thats very interesting, thank you. I've taken a look at your seed site link and can see what you mean.
              My 'close window' clicky is the one that Actinic has put there. Under normal catalog operation the pop-up is generated in a window of set size by the javascript 'MORE' button and pressing 'Close Window' takes you back to where you were. I didn't realise the close window actually re-directed the customer to the page where they were rather than simply closing the window.... I wonder why Actinic did it this way?
              It seems that the close window text does exactly what I have written my red text to do! But in Java not html.... I think I'll keep them both there though as the customer using the site as Actinic intended it would be inclined to push 'Close Window' and anyone getting to that page via a search engine would probably be looking for a message like the one in red text
              Thanks for taking the time to look at my site and write your comments Dave.
              Regards
              Richard
              http://www.turbobits.co.uk

              Comment

              Working...
              X