robots.txt and Environments
- Enable robots.txt
- How the environment changes the output
- Deploy previews
- Testing the generated file
- Overriding the template
Ananke ships a robots.txt template that changes its output depending on the
build environment, so production sites are crawlable while previews and local
builds are not.
Enable robots.txt #
Tell Hugo to generate robots.txt by enabling it in your configuration:
# config/_default/hugo.toml
enableRobotsTXT = trueWith this on, Hugo renders the theme’s template to public/robots.txt on every
build.
How the environment changes the output #
The template checks whether the build is a production build:
Production (
hugowith the default environment, orhugo --environment production) allows crawling and points to the sitemap:User-agent: * Allow: / Sitemap: https://example.com/sitemap.xmlAny non-production build (for example
hugo server, or--environment development) disallows everything:User-agent: * Disallow: /
This mirrors the per-page indexing behaviour described in SEO → Content indexing: non-production output is kept out of search engines.
Deploy previews #
Branch and pull-request previews (for example Netlify deploy previews) should not be indexed. Build them in a non-production environment so the disallow output is produced. Netlify sets a non-production context for previews automatically; if you build previews yourself, pass a non-production environment:
hugo --environment developmentOnly your live production deploy should run in the production environment.
Testing the generated file #
Build for production and inspect the output:
hugo --environment production
cat public/robots.txtYou should see Allow: / and the Sitemap: line. Then build without the
production environment and confirm it switches to Disallow: /:
hugo --environment development
cat public/robots.txtOverriding the template #
To customise the rules, override the theme’s template by creating
layouts/robots.txt in your own project. Hugo uses your copy instead of the
theme’s. Keep the environment check if you still want previews excluded:
User-agent: *
{{ if hugo.IsProduction -}}
Allow: /
Sitemap: {{ "/sitemap.xml" | urls.AbsURL }}
{{ else -}}
Disallow: /
{{ end -}}