robots.txt Is a Contract: Parsing It Without Lying to Yourself

You fetched /robots.txt, got a 200, grepped for Disallow, found nothing scary, and started crawling....