Sitemap & robots.txt Generation

Automatic Sitemaps

Export a default function from app/sitemap.ts that returns an array of URL objects. Next.js serves it as /sitemap.xml. Use generateSitemaps() to split large sites across multiple files.

Sitemap & robots.txt Generation

Search engine optimization (SEO) isn't just about writing meta tags; it's about guiding search engine crawlers through your application efficiently. If Googlebot wastefully crawls non-public dashboards or misses newly published articles, your search rankings suffer. By generating standard, dynamic sitemap.xml and robots.txt files, you tell crawlers exactly where to direct their crawl budget, ensuring search engines index your key content instantly.

---

1. How Next.js 15 Handles Special SEO Files

In previous versions of Next.js, developer-managed sitemaps were generated through custom build scripts or Route Handlers that output XML headers manually. The Next.js App Router simplifies this. By placing a file named sitemap.ts (or .js) and robots.ts (or .js) directly in the root of the app/ folder, Next.js handles XML compilation, headers, and caching mechanisms automatically.

These files are treated as dynamic server-side assets. Next.js processes them and outputs them as standard public routes (e.g., /sitemap.xml and /robots.txt) with correct MIME types automatically.

---

2. Configuring robots.txt with app/robots.ts

The robots.txt file acts as a gatekeeper, instructing crawlers which paths are safe to index. In Next.js, we define this programmatically by exporting a default function that returns a Robots configuration object.

Here is a standard configuration that blocks private routes while permitting general crawling and explicitly linking the sitemap:

// app/robots.ts
import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: [
          '/admin/',
          '/dashboard/',
          '/api/',
          '/*?page=*', // Prevent crawling redundant query parameters
        ],
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

---

3. Generating a Static Sitemap Using app/sitemap.ts

For simple static websites, you can declare your application routes directly within app/sitemap.ts. The default export function must return an array of sitemap objects.

Here is how to structure a basic static sitemap:

// app/sitemap.ts
import { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: `${baseUrl}/about`,
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    {
      url: `${baseUrl}/courses`,
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.9,
    },
  ];
}

---

4. Dynamic Sitemaps with Database Integration

In full-stack Next.js applications, your routes aren't static. You have dynamic blogs, products, or course lessons stored in a database. To ensure these dynamic pages are crawled, fetch your records directly inside sitemap.ts.

Because sitemap.ts runs on the server, you can query your database using Prisma, Drizzle, or raw fetch requests securely.

// app/sitemap.ts
import { MetadataRoute } from 'next';

interface Course {
  slug: string;
  updatedAt: string | Date;
}

async function getCourses(): Promise<Course[]> {
  // Query database or API endpoint
  const res = await fetch('https://api.progrumar.com/courses', {
    next: { revalidate: 3600 } // Cache results for 1 hour
  });
  return res.json();
}

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';
  const courses = await getCourses();

  const courseUrls = courses.map((course) => ({
    url: `${baseUrl}/courses/${course.slug}`,
    lastModified: new Date(course.updatedAt),
    changeFrequency: 'weekly' as const,
    priority: 0.8,
  }));

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1.0,
    },
    ...courseUrls,
  ];
}

---

5. Fine-Tuning Sitemap Fields (lastModified, changeFrequency, priority)

Search engines look at sitemap properties to schedule how often they recrawl your pages. Fine-tuning these values helps optimize crawl efficiency:

Property	Data Type	SEO Best Practice
lastModified	Date | string	Provide the exact date/time the page content was last updated. Do not simply return new Date() for every page on every request, as crawlers will ignore it if it doesn't match reality.
changeFrequency	'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never'	Align with the real rate of updates. Use 'daily' for active listings or blogs, and 'monthly'/'yearly' for static about pages.
priority	number (0.0 to 1.0)	Set relative priorities. Landing pages get 1.0, main list pages get 0.8, and individual articles or legal pages get 0.5. Note that Google ignores this, but other search engines still read it.

---

6. Internationalization and Localized Sitemaps

If your application supports multiple languages, you must inform crawlers about localized versions of your pages using alternate URLs (hreflang metadata).

The Next.js sitemap type supports an alternates configuration block, which maps languages to their localized URL formats:

// app/sitemap.ts
import { MetadataRoute } from 'next';

export default function sitemap(): MetadataRoute.Sitemap {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';

  return [
    {
      url: `${baseUrl}/courses`,
      lastModified: new Date(),
      alternates: {
        languages: {
          es: `${baseUrl}/es/courses`,
          fr: `${baseUrl}/fr/courses`,
          de: `${baseUrl}/de/courses`,
        },
      },
    },
  ];
}

---

7. Handling Scale: Splitting Large Sitemaps

A single sitemap file is limited to 50,000 URLs and a file size of 50MB by search engine protocols. If your website exceeds this (for example, a large e-commerce platform), you must segment your sitemaps.

Next.js 15 provides a built-in function called generateSitemaps(). When exported alongside your default sitemap generation function, Next.js generates indexed sitemaps (e.g., /sitemap/0.xml, /sitemap/1.xml) automatically.

// app/sitemap.ts
import { MetadataRoute } from 'next';

interface SitemapId {
  id: number;
}

// Generate the sitemap IDs that will correspond to sitemaps paths
export async function generateSitemaps(): Promise<SitemapId[]> {
  // Let's assume we have 150,000 items and want to split them into chunks of 50,000
  return [{ id: 0 }, { id: 1 }, { id: 2 }];
}

export default async function sitemap({
  id,
}: {
  id: number;
}): Promise<MetadataRoute.Sitemap> {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';
  
  // Calculate offset based on current index id
  const limit = 50000;
  const offset = id * limit;

  // Fetch target subset of database records
  const items = await fetch(`https://api.progrumar.com/items?limit=${limit}&offset=${offset}`).then(r => r.json());

  return items.map((item: { slug: string; updatedAt: string }) => ({
    url: `${baseUrl}/items/${item.slug}`,
    lastModified: new Date(item.updatedAt),
  }));
}

---

8. Environment-Based robots.txt Rules

You should only allow indexing on your production environment. If you deploy staging, user-testing, or preview branches, search engines might crawl and index duplicate pages, causing a drop in your production SEO domain authority.

You can check your current environment variables inside robots.ts to prevent indexing outside of production:

// app/robots.ts
import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NEXT_PUBLIC_ENV === 'production';
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';

  if (!isProduction) {
    return {
      rules: [
        {
          userAgent: '*',
          disallow: '/', // Block everything on staging/preview
        },
      ],
    };
  }

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin/', '/dashboard/'],
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

---

9. Programmatic Path Exclusion

Often, specific routes should not be exposed to search engines. Rather than hardcoding these exceptions inside robots.ts, you can exclude draft posts or unlisted courses directly inside your dynamic sitemap array loop:

// app/sitemap.ts
import { MetadataRoute } from 'next';

interface PageItem {
  slug: string;
  isPrivate: boolean;
  status: 'draft' | 'published';
  updatedAt: string;
}

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://progrumar.com';
  
  const pages: PageItem[] = await fetch('https://api.progrumar.com/pages').then(r => r.json());

  const publicUrls = pages
    .filter((page) => !page.isPrivate && page.status === 'published')
    .map((page) => ({
      url: `${baseUrl}/pages/${page.slug}`,
      lastModified: new Date(page.updatedAt),
    }));

  return [
    {
      url: baseUrl,
      lastModified: new Date(),
    },
    ...publicUrls,
  ];
}

---

10. Common Gotchas

• Caching Issues: Next.js builds sitemaps statically by default during next build. If your database is updated frequently, make sure to add revalidation boundaries (e.g. exporting export const revalidate = 3600) inside the sitemap file, or change the default caching configuration on your fetch requests.
• Missing Absolute URLs: Sitemaps require absolute URLs (including the protocol, e.g. https://). Relative paths like /courses/nextjs will fail search engine parsers.
• Conflicting Route Handlers: Do not create an app/sitemap.xml/route.ts or public/sitemap.xml alongside app/sitemap.ts. Next.js will encounter naming collision errors, preventing compilation.
• Database Connection Exhaustion: When generating thousands of pages dynamically inside sitemap.ts, make sure to request only the necessary fields (e.g. slug, updatedAt) from your database instead of fetching full text contents, preventing high memory usage during build compilation.

---

Key Takeaways

• Place sitemap.ts and robots.ts directly inside the app/ root directory to configure crawler settings automatically.
• Block crawlers on staging, development, and preview deployments using environment variables inside robots.ts.
• Sitemaps require absolute domain prefixes (always use your production URL base).
• Split sitemaps dynamically using generateSitemaps() when handling databases that exceed 50,000 public paths.
• Utilize the alternates configuration structure to signal language-specific pages to crawlers.

Once your sitemap and robots settings are in place, you need to enrich search engine search results with structured metadata. In the next lesson, we will explore Structured Data (JSON-LD), learning how to inject schema.org formats directly into your React Server Components to render rich snippets and FAQs directly on search engine result pages.