AI crawlers and robots.txt: a practical guide for developers

A surprising number of sites block the exact AI crawlers they would like to be cited by, usually...