llms.txt, robots.txt and AI crawlers: how to make your website readable by AI

Short answer: allow the AI crawlers in robots.txt, serve your key facts as plain HTML text, add...