allow search engines from crawling only on production #15153
Conversation
✅ Deploy Preview for docsdocker ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify site settings. |
|
I think we set a
Looks like we picked the |
| {%- if jekyll.environment == 'production' -%} | ||
| User-agent: * | ||
|
|
||
| # Docker Engine archives |
There was a problem hiding this comment.
Once we have proper redirects, we can probably remove all of these (I think these were added when we were still hosting archives, but they also helped to prevent Google indexing archived URLs it found elsewhere with "new" content, but using the "old" URL - not 100% sure though).
robots.txt
Outdated
| This liquid template is necessary to only allow search engine crawling in | ||
| production environment. We don't want staging or long-lived previews on Netlify | ||
| to be indexed by search engines. |
There was a problem hiding this comment.
Should we output this as a regular comment in robots.txt if indexing is disabled? e.g. something like;
# Disable all indexing on staging websites and Netlify previews to prevent
# them showing up in search results.
robots.txt
Outdated
| {%- else -%} | ||
| User-agent: * | ||
| Disallow: / | ||
| {%- endif -%} |
There was a problem hiding this comment.
Not sure if it's checking for that, but looks like the %- and -% on both sides causes no newline to be present at the end; perhaps (one of them) needs to be removed
Hum yeah I think we should use |
crazy-max
force-pushed
the
fix-robots-txt
branch
from
469e035 to
bda0fcf
Compare
crazy-max
force-pushed
the
fix-robots-txt
branch
from
bda0fcf to
006b24c
Compare
_includes/head.html
Outdated
| <head> | ||
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> | ||
| {%- if page.sitemap == false or site.GH_ENV == "gh_pages" %} | ||
| {%- if jekyll.environment != 'production' %} |
There was a problem hiding this comment.
We probably need to keep the page.sitemap as well; it's used for pages that are not removed, but no longer present in the TOC (and/or shouldn't be indexed, such as pages about deprecated things)
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
crazy-max
force-pushed
the
fix-robots-txt
branch
from
006b24c to
ef40cff
Compare
2 participants
while working on #15151, I found out that crawling for search engines was enabled for staging and netlify previews (pull requests): https://www.google.com/search?q=site%3Adocs-stage.docker.com
Signed-off-by: CrazyMax crazy-max@users.noreply.github.com