XML Sitemaps

[john@stinkyink.]

If you have ever wondered why we bother with generating an XML sitemap, then wonder no more!. Have a look at this link for an interesting (well I thought it was interesting - but they all reckon I need to get out more!) anecdote about misusing and then understand XML Sitemaps

[jont]

Scary how a small change can provide disastrous results... funny how it is easier to kill your site than it is to improve it

[combat68]

xml so important

Geez thats scary!

I'm new to all this and working on getting google to pick up the right pages

So after a couple of days of putting together robots.txt and an accurate xml sitemap i'm looking forward to seeing if google will refresh its indexing where i want it.

Cheers for the info

Rob

[Stereo Steve]

We binned the XML sitemap as it's too easy to get wrong, particularly with the frequency thing. We found google indexed new pages but put them straight into supplemental without indexing the content. They would then stay there for months despite all efforts to link to them from the index or from other sites etc.

Letting google find our new pages naturally from a home page link may take a day or so longer but at least it looks at them properly and most times they go straight to the main index.

I saw no increase in total pages indexed with a sitemap. My advice would be to make sure your internal linking is well structured and not bother. It gives google an excuse to not crawl your site, just index the URL and not bother with the rest.

[Buzby]

John,

Which software do you use to generate your feed?

I use Mole End's mash but it doesn't have any priorities to the urls, just urls.

Kind regards

Jason

[grantglendinnin]

So the moral of the story is....automated is never easy!? I'm looking at the way our site is laid out and now thinking of the nightmare of creating an XML sitemap for the entire site with every product having its own single page

[john@stinkyink.]

We were using G-Site crawler, and just accepted the default settings which appeared to be OK. Then Rob wrote his own version which gave us the problem. The learning curve was vicious, but I think that we are better people for a bit of pain!!!!!!!

I'm quite relaxed about it now, once we understood what had happened and how to benefit from it then we can use the knowledge. Not understanding how the Sitemap is being used is the bigger issue I think. Having set the priorities as we indicated in the Figure, the program now Auto updates at midnight every night and then pings Google to tell it if anything has changed

Anyway, as Jont said earlier, it is easier to break a website than improve it - but you know us blokes, can't stop tinkering! If it is working fine try to break it!

The thing that scared me was the speed that it all happened at, it was literally dead in hours, and when corrected fixed in hours - which is not how Google usually works

[Mike Hughes]

Intersting stuff John. It looks like something I'll leave alone as it's too complex by far.

What are your thoughts on why the change to the 'priority' value affected the serps so much? Do you think Google just doesn't like seeing a value different from it's calculations or is that value somehow being used in determining the serps? This kind of implies that there could be scope for adjusting how Google ranks the focus of the website.

Mike

PS. The link to Google Webmaster Guidelines doesn't work.

[john@stinkyink.]

Morning Mike,

I don't think its complex once we understood that everything is relative to your index page which is 1. When we dropped the Priority of our printer pages to 0.4 then google just re-ranked them (is that a proper word?) to fairly unimportant in the index - we literally just dropped from front page to nowhere. When we put them back to 0.9 we were almost instantly back in the to their old positions. What I haven't said in the article is that we then (can't resist tinkering) adjusted our product pages to 0.5 to intentionally reduce their importance and increase the importance of the printer pages. That is more difficult to quantify in terms of the difference it has made to our traffic, but we are happy with it at the moment, in fact our Google traffic is increasing at our traditionally quiet time of year.

As we think we understand it, you can't increase your overall importance of your site, but RELATIVE to the importance of your index page you can adjust how Google ranks the rest of your site. That is I think where we are at.

I've searched and there is very little information out there about XML sitemaps so we are offering our experience for what it is worth, if anybody else has any experience of 'Sitemap engineering' (hey, I've invented a new discipline) I would be fascinated to discuss it

(I've fixed the Google link - thanks)

[malbro]

I've searched and there is very little information out there about XML sitemaps so we are offering our experience for what it is worth, if anybody else has any experience of 'Sitemap engineering' (hey, I've invented a new discipline) I would be fascinated to discuss it

(I've fixed the Google link - thanks)

I found this on google webmaster tools for the google version of the sitemap generator, which has to run on the web site host and uses the local server logs to get information.

The Sitemap Generator assigns priority to URLs it finds in the logs based on how often each URL is accessed. For instance, a URL that has been accessed 100 times will be given a higher priority than a URL that has been accessed twice. The actual priority assignment is relative and depends on each URL as compared to other URLs in the site.

[john@stinkyink.]

That's interesting and bears out what we thought. What was striking about the Google auto generator is that it ranks pages on the amount of traffic, so if you prioritise your pages as we have done will Google assume that those pages are the most important?. That is certainly what we think is happening

[Ghome1971]

I uploaded a new site map on 28/06 (G Crawler). My business has nose dived over the past few days but I am not sure if it is just the time of year and current market. I have uploaded a mole end site map which is what I had before just in case. I will keep you posted.

Best Wishes

Geraldine

[Buzby]

I uploaded a more in-depth sitemap about a week ago and have seen a large increase in traffic.

I used the mole end feed and then added the extra information using a find and replace option in Dreaweaver, and then edited the priorities individually.

I would not under any circumstances give Google access to my server or logs just as I would not let a competitor look at my order book.

Google can rank my site on content and not how busy my site is.

A restaurant when quiet fills the window area to give the appearance it is busy. They wouldn't hang a sign on the door saying that they are empty so why actually send that sign to Google by letting them in to view.

[Jan]

That is really interesting John.

I'll be updating the mole end generator to allow additional pages to be defined and to include suplementary pages such as additional info pages. When I do this I will also add the option to define priority, so you can have a default one for most pages but then add a variable to the key ones that assigns a higher one to it. It will be a little while in the making but will be a good addition when it arrives.

Regards,

[Darren B]

Jan I was thinking about you while reading the thread, using a variable to set priorities, i like the idea

John thanks for extremely well laid out case study, and much more i thank you for sharing this sort of information

D