Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.
DEV Community
Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.
114 sites share one robots.txt template: 31 banned agents, half the bytes are a legal notice. Probing 7 of them found two false bans and one 520.
0 comments
No comments yet.