โ‰ˆ the.bay.news

Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.

DEV Community
Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.
114 sites share one robots.txt template: 31 banned agents, half the bytes are a legal notice. Probing 7 of them found two false bans and one 520.

0 comments

Sign in to join the discussion โ€” your thebay.events account works here.

No comments yet.