We let crawlers fetch the pages we never want indexed, because a Disallowed URL never reads its noindex
DEV Community
We let crawlers fetch the pages we never want indexed, because a Disallowed URL never reads its noindex
Munchable's robots.txt disallows two paths and allows three pages we do not want in the index. The fetch is the only thing that can deliver a noindex, and a preview deploy has to disagree with nothing.
0 comments
No comments yet.