Webflow Sitemap and Robots.txt: Default Behavior and How to Override

Webflow generates a sitemap and robots.txt automatically. For most sites, the defaults are fine. But there are specific situations — staging sites, noindexed pages, custom crawl directives — where you need to understand the defaults and know how to override them.

This guide covers what Webflow generates by default, how to control indexing per page and per CMS item, and how to write a custom robots.txt when the auto-generated one isn't enough.

The Auto-Generated Sitemap

Webflow publishes /sitemap.xml automatically when you publish to a custom domain. The sitemap includes:

• All published static pages (unless noindexed)
• All published CMS collection items (unless noindexed)
• The lastmod date for each URL, pulled from Webflow's last-published timestamp

What it doesn't include:

• Utility pages (404, password protection pages)
• Pages marked as "Exclude from search results" in Page Settings
• CMS items marked as noindex
• Pages that are password-protected
• Any URL not part of your Webflow-hosted content (subdomains handled elsewhere, third-party tools)

The sitemap updates every time you publish. There's no delay, no cron job, no manual generation step. Publish in Webflow → sitemap is current.

Controlling Which Pages Are in the Sitemap

For static pages:

Go to Page Settings → SEO tab → check "Exclude this page from search results."

This adds <meta name="robots" content="noindex"> to the page AND removes it from the sitemap. You don't control these independently in Webflow's native UI — noindex and sitemap exclusion are tied together.

For CMS collection items:

You can add a Switch field to your collection called "No Index" (or whatever you want to call it). Then in the collection template's Page Settings → SEO tab, bind the "Exclude this page from search results" toggle to that Switch field.

When the Switch is on for a collection item, that item's page is noindexed and removed from the sitemap. When it's off, it's included.

This is the correct way to handle CMS items you don't want indexed — things like draft posts you've published by accident, thin pages that aren't ready, or pages you want to keep accessible via direct URL but not discoverable via search.

Staging Sites Are Noindexed by Default

Your *.webflow.io subdomain is noindexed. Webflow adds <meta name="robots" content="noindex"> to all pages on staging domains. This is correct behavior — you don't want your staging site competing with your production site in search results.

Webflow also restricts the staging sitemap — it doesn't serve an accessible sitemap.xml on the *.webflow.io domain. Validators and crawlers that expect a sitemap will get a 404 on staging. This is expected.

The Auto-Generated Robots.txt

Webflow generates a default robots.txt at yourdomain.com/robots.txt. The default content is minimal:

User-agent: *
Disallow:

Sitemap: https://yourdomain.com/sitemap.xml

This allows all crawlers to access all pages (an empty Disallow: means no paths are blocked) and references your sitemap. For most sites, this is exactly right.

The key things the default does not include:

• Specific Disallow rules for sensitive paths
• Crawl-delay directives
• Rules targeting specific bots (Googlebot, AdsBot, etc.)
• Multiple sitemap references

Writing a Custom Robots.txt

If you need custom rules, Webflow lets you override the auto-generated robots.txt:

Where: Site Settings → SEO → Indexing → Custom Robots.txt

Type your custom robots.txt content here and publish. Webflow will serve your custom version instead of the default. Your sitemap reference is not automatically preserved — include it explicitly in your custom content.

Example custom robots.txt for a site that wants to block search results pages:

User-agent: *
Disallow: /search
Disallow: /search/

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Example with a crawl delay for aggressive crawlers:

User-agent: *
Crawl-delay: 10

User-agent: Googlebot
Crawl-delay: 0

Sitemap: https://yourdomain.com/sitemap.xml

Note: Googlebot ignores Crawl-delay. It's respected by some other crawlers but not Google. If you're trying to reduce server load from Googlebot specifically, use Google Search Console's crawl rate settings.

What to Check After Launch

When you publish a new Webflow site to a custom domain, verify these before considering indexing set up:

1. Sitemap is accessible and populated:

https://yourdomain.com/sitemap.xml

View it in your browser. Confirm your important pages and CMS items are listed. If the sitemap is empty or missing, the site may not be fully published.

2. Robots.txt is accessible:

https://yourdomain.com/robots.txt

Confirm it has Disallow: (empty, meaning allow all) or your custom rules. Confirm the Sitemap line is present.

3. No important pages are noindexed by accident:

Check the page source (Ctrl+U) of your homepage and key landing pages. Search for noindex. If you find <meta name="robots" content="noindex"> on a page you want indexed, go to Page Settings → SEO tab and uncheck "Exclude this page from search results."

4. Submit to Search Console:

Go to Search Console → Sitemaps → Submit https://yourdomain.com/sitemap.xml. Then use URL Inspection on your most important pages to request indexing.

When the Sitemap Doesn't Update

Webflow updates the sitemap when you publish. If you add a new CMS item and publish it but it doesn't appear in your sitemap, try:

1. Publishing again — sometimes a second publish triggers the sitemap regeneration
2. Waiting 5–10 minutes — sitemap generation can lag slightly after a publish
3. Checking the item isn't accidentally noindexed (CMS switch field)
4. Checking that the collection itself is included in your sitemap settings

The SEO for Webflow guide covers the full technical foundation. If you're working through a site launch, the SEO checklist for Webflow sites has a specific section on sitemap and indexing with the exact Page Settings locations.