the.bay.news

Your robots.txt probably does not block the AI crawler you think it blocks

DEV Community
Your robots.txt probably does not block the AI crawler you think it blocks
A quirk of RFC 9309 means a matched user-agent group inherits nothing from the wildcard group. I found it while cataloguing 42 AI-discovery projects for one ecommerce platform, and the distribution of those 42 says something too.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.