I published an open dataset of every AI crawler user agent (GPTBot, ClaudeBot + 26 more)

DEV Community

I published an open dataset of every AI crawler user agent (GPTBot, ClaudeBot + 26 more)

28 AI crawlers with operator, purpose, and observed robots.txt compliance — JSON + CSV, CC BY 4.0, no signup. Here's why the "purpose" field matters more than the list itself.

28 AI crawlers with operator, purpose, and observed robots.txt compliance — JSON + CSV, CC BY 4.0, no signup. Here's why the "purpose" field matters more than the list itself.