Introducing Google Sitemaps

By Stas Bekman.

Published: June 25th 2006

Some time ago Google has added yet another great tool to its toolkit. And of course it's still in Beta. It's called Google Sitemaps.

It'll probably do much more in the future but at the moment the most important feature is the ability to tell the google crawlers what pages to crawl, how important their are and when Google should revisit your site as you add new changes. The other very useful set of features is the insider view on how google sees your site, whether it sees some problems with it -- e.g., whether it has problems crawling your pages, statistics -- e.g., what keywords people used to find your site and the page rank of each of those pages.

Like all the other tools from Google, this tool is free. All you need to do is to prepare a file containing a listing of the URLs you want to be indexed and their priority, place it at the root directory of your site and you are done. In the future you just need to update that file and Google will pick up the changes from there.

To get started you first need to verify that you are the owner of the site. Rather than using the usual approach of email verification, which is easy to fake, Google asks you to add a new file at your site or add a special tag to the top level index page.

Once verified you use one of the available tools to create a special file sitemaps.xml, upload it to your site and tell Google that it's there. At the moment Google processes the file rather quickly. I imagine that as this service becomes more popular there will be a longer delay. Once Google processed your file, it's going to tell you a lot of information about your site.

Let's start with Diagnostics. Google is going to tell you whether the crawler had any problems reaching certain pages and if it encountered such problems it'll tell you the details, such as HTTP errors, Not found, URLs not followed, URLs restricted by robots.txt, URLs timed out, Unreachable URLs, etc. So that you can take an action and fix those problems on your site.

The second set of features is Statistics, where you can find out various information collected over the period of last 3 weeks. It includes:

• Top search queries (queries on the selected search property that most often returned pages from your site)
• Top search query clicks  (top queries that directed traffic to your site, based on the number of clicks to your pages).
• Average top position (the highest position any page from your site ranked for that query)

Further you can view that information across all Google search engine sites, or only specific ones (e.g., only google.co.uk), so you can tell how high are the chances that it'll be found by visitors from different countries.

It'll then show you crawler statistics, including how many pages have high, medium and lower ranks and how many pages have no rank assigned yet. It'll also show you pages with the highest rank.

It'll then allow you to see the page analysis information as it's seen by Googlebot. It'll show you the common words in your site's content and in external links to your site.

Finally it aggregates all the information that's already available via Advanced Search (Indexed pages, referrers, links, etc.). For example for my site you can see: related:stason.org, allinurl:stason.org, link:stason.org, cache:stason.org, info:stason.org, related:stason.org.

By deploying the information provided by Google Sitemaps you should be able to improve the ranking of your site, by eliminating errors and having more pages indexed, telling Google when you've new pages added, so it will find them faster. All these and other features discussed in this article will help you bring more visitors to your site.

Related Articles