Cross-submissions via robots.txt on Sitemaps.org

Thursday, February 28, 2008

Last spring, the Sitemaps protocol was expanded to include the
autodiscovery of Sitemaps using robots.txt
to let us and other search engines supporting the protocol know about your Sitemaps. We
subsequently also announced support for
Sitemap cross-submissions using Google Webmaster Tools,
making it possible to submit Sitemaps for multiple hosts on a single dedicated host. So it was
only time before we took the next logical step of marrying the two and allowing Sitemap
cross-submissions using robots.txt. And today we’re doing just that.

We’re making it
easier for webmasters to place Sitemaps for multiple hosts on a single host
and then letting us know by including the location of these Sitemaps in the appropriate
robots.txt.

How would this work? Say for example you want to submit a Sitemap for each of the two hosts you
own, www.example.com and host2.google.com. For simplicity’s sake, you
may want to host the Sitemaps on one of the hosts, www.example.com. For example, if
you have a Content Management System (CMS), it might be easier for you to change your robots.txt
files than to change content in a directory.

You can now exercise the cross-submission support via robots.txt (by letting us know the location
of the Sitemaps):

1. The robots.txt for www.example.com would include:
2. And similarly, the robots.txt for host2.google.com would include:

By indicating in each individual host’s robots.txt file where that host’s Sitemap lives you are
in essence proving that you own the host for which you are specifying the Sitemap. And by
choosing to host all of the Sitemaps on a single host, it becomes simpler to manage your Sitemaps.

We are making this announcement today on
Sitemaps.org as a joint
effort. To see what our colleagues have to say, you can also check out the blog posts published
by Yahoo! and
Microsoft.

Posted by Prashanth Koppula, Product Manager