A brand hands us a domain, and most of the crawler is about which five pages not to read

Six regexes pick the key pages, one Set stops the nav being read five times, the sitemap is sampled round robin across sections, and eight failure kinds each map to a sentence the customer can act on.