Every page we serve advertises an RSS feed our own robots.txt forbids
DEV Community
Every page we serve advertises an RSS feed our own robots.txt forbids
I wrote a robots.txt matcher and ran all 530 URLs from our sitemap through our 850-line rules file. The sitemap passed. The URLs that are not in the sitemap were the interesting part.
0 comments
No comments yet.