≈ the.bay.news

We let crawlers fetch the pages we never want indexed, because a Disallowed URL never reads its noindex

DEV Community
We let crawlers fetch the pages we never want indexed, because a Disallowed URL never reads its noindex
Munchable's robots.txt disallows two paths and allows three pages we do not want in the index. The fetch is the only thing that can deliver a noindex, and a preview deploy has to disagree with nothing.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.