I need to put in place a system that will generate sitemap files (https://www.sitemaps.org) for a huge website with content changing dynamically. These are example figures, but I'm thinking of something at and above these orders of magnitude:
My ongoing sitemap goals after search engines index everything initially are:
I'll add that I suspect sites such as Wikipedia or StackOverflow are in a similar position.
Are there any good algorithms for such a use case?