Extract every URL from a sitemap.xml in Python (indexes, .gz, robots.txt) and spot competitors' new pages

A ~80-line Python script that finds a site's sitemaps via robots.txt, follows sitemap indexes, reads gzipped sitemaps and tells you which URLs are new since the last run. Tested on real sites in October 2026.