Inside Google Sitemaps: August 2005
Your source for product news and developments
If you've created and submitted Sitemaps for your non-mobile pages, or just want to submit a mobile Sitemap for the first time, here are a few helpful tips to help you get started:
Identify your mobile Sitemaps content
We just launched two new features.
You can already use Google Mobile Web Search on your mobile phone to search through sites that have been specifically designed for mobile phones, PDAs, and other handheld devices. We add new sites to our mobile web index regularly.
You can help users find your mobile webpages by letting us know about those pages. Google Mobile Sitemaps lets you submit Sitemaps for URLs that serve mobile content. You create and submit a mobile Sitemap much in the same way you do other Sitemaps: with the Sitemap Generator, the Sitemap protocol, or via a syndication feed or text file.
The biggest difference with a mobile Sitemap is that you have to submit a separate one for each markup language. Right now we support:
New Site Statistics
Now you can see information for the URLs in your site that we've had trouble accessing – both for URLs from your Sitemap and those we've discovered during a regular crawl. We won't show you these additional statistics until you verify your site, which is a very simple process. Click the verify link next to the site on your My Sitemap page, create an empty file using the name we specify, and upload it to the folder where your Sitemap is located. We'll check to see that the file is there, which tells us that you have permission to upload files to that site, and then we'll show you the information.
So, what URLs from your site should you put in your Sitemap?
Put in everything! List the URLs that contain your content, images, media, and anything else in your site.
If you want to include only a subset of items, you can, but we’d like as much information about your site as you can give us.
Remember that we respect robots.txt, so if you include any URLs in your Sitemap that are restricted in robots.txt, we won’t crawl those.
How should you name your Sitemap? What extension should you give it?
The short answer is that you can name your Sitemap anything you want. You can use any extension. Just submit the URL to us, and we’ll go pick it up.
The better answer is a little longer. We recommend that you give your Sitemap an extension that identifies the file type. For instance, if you create a simple text file that lists URLs, we suggest giving that Sitemap a .txt extension.
If you create an XML Sitemap that uses our Sitemap protocol, give it an .xml extension. If you compress that file using gzip, give it an .xml.gz extension.
If you use our Sitemap Generator to create a Sitemap, you specify the resulting Sitemap name in the config.xml file. The default name is sitemap.xml.gz. If you keep the .gz extension, the resulting Sitemap file will be compressed. If you change this name to have an .xml extension, the resulting file will not be compressed. We suggest you compress the file so that your webserver will take less of a bandwidth hit when we download it.
You can submit the URL of a script that dynamically generates an XML Sitemap when we download it. That script might have an extension such as .asp or .php (depending on the script type). The extension of the file isn’t a problem, but if your script takes a long time to run, the delay will look like a server timeout and we’ll try again later. If you have trouble getting this type of Sitemap submitted, make sure your script is responsive. Also ensure that your webserver doesn’t automatically add things (such as HTML headers and footers) to the generated files, since that would cause the resulting XML file to have parsing errors.
One more thing about naming. You can name your Sitemap anything you want… almost. You can’t name it robots.txt. And if you use a robots.txt file for your site, make sure that it doesn’t restrict our access to your Sitemap file.
Several of you have asked us, Should I submit Sitemaps or Sitemap indexes for my site?
If you have a small site, you probably don't need to use a Sitemap index file -- you can just list all of your URLs in one Sitemap.
If you have a larger site, you may want or need to have multiple Sitemaps for your site. In that case, you can make submitting and tracking easier by listing the Sitemaps in a Sitemap index file.
You must use multiple Sitemaps for your site when:
You can also have an index of Sitemap index files. A Sitemap index file can be a maximum of 10MB as well, so if you have a really large site, you may have to use this additional organization step to keep the file sizes to a manageable level. We have a size limitation for Sitemaps and Sitemap indexes so that when we download the files, we don't overwhelm your bandwidth.
Compressing your Sitemap index file
Speaking of being considerate of your bandwidth, if you can, you should compress your Sitemaps and your Sitemap indexes using gzip. If you're not familiar with gzip, keep watching this blog. We're putting together some helpful instructions.
If you compress your Sitemap index file, you'll probably want to give it an .xml.gz extension. If you don't compress your Sitemap index file, you'll probably want to give it an .xml extension.
Submitting your Sitemap index file
So, you've got some individual Sitemaps that you've listed in a Sitemap index file. What now? Just Sign into Google Sitemaps and submit the Sitemap index file. You don't need to submit individual Sitemaps that are included in the index. Once we've processed your Sitemap index file, we'll let you know if we found errors in the Sitemap index itself, or in any of the individual Sitemaps.
If you make changes to a Sitemap included in a Sitemap index file you've submitted, just change the lastmod date for that Sitemap in your index. During this beta period, feel free to resubmit the Sitemap index file.
We are stoked that so many of you are trying out Sitemaps while it's in beta. When we read on the Sitemaps Google Group that you'd like an easy way to find out about new features and where we're headed with this thing, we realized we could put our very own Blogger to good use.
And since the Sitemaps team is far-flung in Mountain View, Kirkland, and Zurich, this will be a good way to keep everyone posted about new features and developments. Subscribe to our feed to keep up with the latest. We'll also e-mail each blog post to the Sitemaps Google Group, so if you get your news from there, you won't miss out on a thing.
In addition to telling you about new features here, we'll also do our best to address some of the frequently asked questions from the Google Group.
Lately, some of you have wondered if submitting a Sitemap could actually reduce the number of your site pages we have indexed.
No, submitting a Sitemap will not reduce the number of indexed pages for your site.
As we note in our information for Webmasters, each time we update our database of webpages, our index shifts: we find new sites, we lose some sites, and sites' rankings change. If your site was dropped from Google and you haven't made major changes to it, we'll likely pick it up again soon. If your site's ranking changes, ensure you are following our guidelines.
When you submit your Sitemap, you help us learn more about the contents of your site. Participation in this program will not affect your pages' rankings or cause your pages to be removed from our index.
Copyright © 2005 Google Inc. All rights reserved.