sitemaps.xml: Allow gzip support, cache?

[smagnusson]

Google's suggestion in the sitemaps protocol is to provide it as a .gz file, and not just as an .xml file carried over mod_deflate.

Enable support for accessing /sitemap.gz (or is sitemap.xml.gz?)

This will increase the CPU load for producing the sitemap, and so it would be best for this to be cached, say to /assets/.sitemap.xml.gz

E.g.

if (cachefile < 1 hour old OR ?flush=1 ) create-xml-file-and-save-to-gzipped-cache-file()

header temporary redirect to cache file()

Does the protocol support putting a comment inside the xml file to say the generation time?

[wscott]

The Google document on the specification is here:

Sitemap Protocol

Google says you can compress your Sitemap, but doesn't give any benefit to doing so. The max size restriction on the file is applied to the uncompressed version. Additionally, a single Sitemap file will not be regularly accessed. (you will likely modify the file by adding a page to your site more often than it is queried by a search engine.)

[smagnusson]

Will Scott mentions:

The sitemap isn't a gz file, but that shouldn't be an issue. In the initial negotiation with Apache, Google's servers will say they can handle gzip-compressed data, so apache ought to send out a gzip compressed version of the data. (to my knowledge serving the file as .gz will be beneficial only if you have the gzip php module, and have content negotiation turned off in apache.)

The file is also not cached. And again I'm not sure how much benefit would be gained by caching it. there are only a limited number of search engines that actually check the sitemap file, and they check it very infrequently. Silverstripe is set up to ping Google when it's updated, and that's going to happen more often even than Google will get around to checking the file. The file will probably only get requested a couple times per version at most, so it's unlikely to need to get cached.

Sig concludes:

Lets do this anyway, at some point, to prevent a DOS which could occur if someone accesssed the file over and over again.

First of all we will see how slow it is, tomorrow I will benchmark one, and I will close or keep the issue as appropriate.

[smagnusson]

Page performance is alright for the moment, but it will need to cache so that as sites grow and become popular, they won't be brought to their knees by continuous requests to sitemap.xml

Benchmark: 2 seconds for a 350 page website (www.silverstripe.com) on a Pentium 4 2.6ghz with 1.5 GB ram.

Also as the site grows to 1000+ pages, I assume the memory needs of the website will be getting very large.