robots.txt and Environments

• Enable robots.txt
• How the environment changes the output
• Deploy previews
• Testing the generated file
• Overriding the template

Ananke ships a robots.txt template that changes its output depending on the build environment, so production sites are crawlable while previews and local builds are not.

Enable robots.txt #

Tell Hugo to generate robots.txt by enabling it in your configuration:

# config/_default/hugo.toml

enableRobotsTXT = true

With this on, Hugo renders the theme’s template to public/robots.txt on every build.

How the environment changes the output #

The template checks whether the build is a production build:

• Production (hugo with the default environment, or hugo --environment production) allows crawling and points to the sitemap:

  Copy

  User-agent: *
  Allow: /
  Sitemap: https://example.com/sitemap.xml

• Any non-production build (for example hugo server, or --environment development) disallows everything:

  Copy

  User-agent: *
  Disallow: /

This mirrors the per-page indexing behaviour described in SEO → Content indexing: non-production output is kept out of search engines.

Deploy previews #

Branch and pull-request previews (for example Netlify deploy previews) should not be indexed. Build them in a non-production environment so the disallow output is produced. Netlify sets a non-production context for previews automatically; if you build previews yourself, pass a non-production environment:

hugo --environment development

Only your live production deploy should run in the production environment.

Testing the generated file #

Build for production and inspect the output:

hugo --environment production
cat public/robots.txt

You should see Allow: / and the Sitemap: line. Then build without the production environment and confirm it switches to Disallow: /:

hugo --environment development
cat public/robots.txt

Overriding the template #

To customise the rules, override the theme’s template by creating layouts/robots.txt in your own project. Hugo uses your copy instead of the theme’s. Keep the environment check if you still want previews excluded:

User-agent: *
{{ if hugo.IsProduction -}}
Allow: /
Sitemap: {{ "/sitemap.xml" | urls.AbsURL }}
{{ else -}}
Disallow: /
{{ end -}}