A web crawler for AI agents with no LLM in the crawl loop

When an AI agent needs an answer that lives a few clicks deep on a website, there are two common ways...