Next.js Sitemap and Robots.txt with the App Router

The App Router ships with typed file conventions for both sitemap.xml and robots.txt. You create app/sitemap.ts and app/robots.ts, export a function from each, and Next.js serves the correct content type at the right URL. No packages, no static files to keep in sync.

Here's the production pattern for both.

app/sitemap.ts

Create app/sitemap.ts and export a default function that returns a MetadataRoute.Sitemap array:

// app/sitemap.ts
import type { MetadataRoute } from 'next'

export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://yourdomain.com',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 1,
    },
    {
      url: 'https://yourdomain.com/about',
      lastModified: new Date('2026-01-01'),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    {
      url: 'https://yourdomain.com/pricing',
      lastModified: new Date('2026-01-01'),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
  ]
}

Next.js serves this at /sitemap.xml with the correct application/xml content type.

Dynamic Sitemap from a Database

The real value is pulling entries from your database. Make the function async:

// app/sitemap.ts
import type { MetadataRoute } from 'next'

async function getPosts() {
  const res = await fetch(`${process.env.API_URL}/posts?fields=slug,updatedAt`, {
    next: { revalidate: 3600 },
  })
  return res.json() as Promise<Array<{ slug: string; updatedAt: string }>>
}

async function getProducts() {
  const res = await fetch(`${process.env.API_URL}/products?fields=slug,updatedAt`, {
    next: { revalidate: 3600 },
  })
  return res.json() as Promise<Array<{ slug: string; updatedAt: string }>>
}

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const [posts, products] = await Promise.all([getPosts(), getProducts()])

  const staticRoutes: MetadataRoute.Sitemap = [
    {
      url: 'https://yourdomain.com',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 1,
    },
    {
      url: 'https://yourdomain.com/blog',
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
  ]

  const postRoutes: MetadataRoute.Sitemap = posts.map((post) => ({
    url: `https://yourdomain.com/blog/${post.slug}`,
    lastModified: new Date(post.updatedAt),
    changeFrequency: 'weekly',
    priority: 0.7,
  }))

  const productRoutes: MetadataRoute.Sitemap = products.map((product) => ({
    url: `https://yourdomain.com/products/${product.slug}`,
    lastModified: new Date(product.updatedAt),
    changeFrequency: 'monthly',
    priority: 0.6,
  }))

  return [...staticRoutes, ...postRoutes, ...productRoutes]
}

Splitting Large Sitemaps

Google's sitemap limit is 50,000 URLs or 50MB per file. For large sites, split into multiple sitemaps. The App Router supports this via multiple sitemap files using route segments:

app/
  sitemap.ts           → /sitemap.xml (sitemap index)
  blog-sitemap.ts      ← won't work — only sitemap.ts is the convention

For split sitemaps, use the generateSitemaps export alongside your default function:

// app/sitemap.ts
import type { MetadataRoute } from 'next'

const POSTS_PER_SITEMAP = 1000

export async function generateSitemaps() {
  const totalPosts = await fetch(`${process.env.API_URL}/posts/count`)
    .then((r) => r.json())
    .then((d) => d.count as number)

  const sitemapCount = Math.ceil(totalPosts / POSTS_PER_SITEMAP)

  return Array.from({ length: sitemapCount }, (_, i) => ({ id: i }))
}

export default async function sitemap({
  id,
}: {
  id: number
}): Promise<MetadataRoute.Sitemap> {
  const posts = await fetch(
    `${process.env.API_URL}/posts?offset=${id * POSTS_PER_SITEMAP}&limit=${POSTS_PER_SITEMAP}`
  ).then((r) => r.json())

  return posts.map((post: { slug: string; updatedAt: string }) => ({
    url: `https://yourdomain.com/blog/${post.slug}`,
    lastModified: new Date(post.updatedAt),
    changeFrequency: 'weekly' as const,
    priority: 0.7,
  }))
}

With generateSitemaps, Next.js serves sitemap shards at /sitemap/0.xml, /sitemap/1.xml, etc., and automatically generates a sitemap index at /sitemap.xml.

app/robots.ts

Create app/robots.ts and export a default function returning a MetadataRoute.Robots object:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin/', '/api/', '/private/'],
    },
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

This generates:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/

Sitemap: https://yourdomain.com/sitemap.xml

Differentiated Rules per Bot

For cases where you want to allow all crawlers but block AI training bots from specific paths:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin/', '/api/private/'],
      },
      {
        userAgent: ['GPTBot', 'Google-Extended', 'CCBot'],
        disallow: '/',
      },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
    host: 'https://yourdomain.com',
  }
}

Environment-Specific robots.ts

During development and on staging environments, you typically want to block all crawlers. Use an environment variable to switch:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NEXT_PUBLIC_ENV === 'production'

  if (!isProduction) {
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    }
  }

  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: ['/admin/', '/api/'],
    },
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Verifying Your Sitemap

After deploying, validate your sitemap structure with RankCrab's sitemap generator — useful for cross-checking that all expected URLs are present and that lastModified dates are correct.

Free tool

XML sitemap generator

Paste URLs, get a valid sitemap.xml you can copy or download.

Try it

Check your robots.txt output against what crawlers will actually see. A misconfigured disallow rule can silently block your entire site from being indexed.

Free tool

Robots.txt generator + tester

Build robots.txt with a form, test any URL against it offline.