crawlberg-robots v1.9.0 — The robots.txt parser shared by the crawlberg crawler and its headless-browser fallback.

Sign in to join the discussion — your thebay.events account works here.