≈ the.bay.news

Every page we serve advertises an RSS feed our own robots.txt forbids

DEV Community
Every page we serve advertises an RSS feed our own robots.txt forbids
I wrote a robots.txt matcher and ran all 530 URLs from our sitemap through our 850-line rules file. The sitemap passed. The URLs that are not in the sitemap were the interesting part.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.