AVAILABLE FOR WEB, FLUTTER & QA PROJECTS

📞+44 7542 174481|💬WhatsApp Chat
Faisal RafiqueSoftware Architect
Start a Project→
FRFaisal Rafique
Start a Project→💬Chat on WhatsApp (+44 7542 174481)
📍 Street 3, Peoples Colony, Vehari, Punjab, Pakistan
App Router robots.ts Code✓ Free & No Login RequiredCluster Hub: Full-Stack Web Development →

Next.js Robots.txt Generator

The Next.js Robots.txt Generator creates type-safe `app/robots.ts` files for the Next.js App Router, managing search crawler permissions, private dashboard route protection, sitemap directives, and AI bot governance.

Disallowed Rules

4 paths

Internal protected routes

Blocked AI Bots

1 bots

Content scrapers restricted

Preview Protection

Active

Blocks Vercel previews

Target Endpoint

/robots.txt

RFC 9309 specification

4 paths entered
AI Crawler & Scraper Governance
Generated src/app/robots.ts FileTypeScript
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://faisalrafique.com';

  // Shield Vercel / staging preview deployments from indexation
  if (process.env.VERCEL_ENV === 'preview' || process.env.NODE_ENV !== 'production') {
    return {
      rules: {
        userAgent: '*',
        disallow: ['/'],
      },
    };
  }

  return {
    rules: [
      {
        userAgent: '*',
        allow: ['/'],
        disallow: [
          '/api/',
          '/dashboard/',
          '/admin/',
          '/login',
        ],
      },
    {
      userAgent: 'Bytespider',
      disallow: ['/'],
    },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

How to use this tool

1. Specify Base Site URL

Enter your production domain (e.g. https://faisalrafique.com).

2. Define Disallowed Paths

List restricted routes such as `/dashboard/`, `/api/`, `/admin/`, or `/private/`.

3. Configure AI Scraper Policies

Choose whether to allow or block AI crawlers (GPTBot, CCBot, ClaudeBot, PerplexityBot).

4. Copy app/robots.ts Code

Paste the generated TypeScript snippet directly into your `src/app/robots.ts` file.

Formula or logic used

Next.js Robots Specification

Next.js translates `app/robots.ts` using `MetadataRoute.Robots` into standard RFC 9309-compliant robots.txt files at compile time.

  • File Location: `app/robots.ts` serves dynamically or statically at `/robots.txt`.
  • Root Sitemaps: Always include absolute URL pointer `sitemap: `${baseUrl}/sitemap.xml`.
  • Wildcards: App Router supports wildcard string arrays for userAgent and disallow paths.
  • Host Directive: Optional Host header declaration for multi-domain deployments.

Examples

Example 1: Standard SaaS Application robots.ts

Input: Allow public pages; disallow /dashboard, /api, /admin; link /sitemap.xml.
Calculated Result: Type-safe MetadataRoute.Robots function returning standard crawler rules.

Shields internal administrative and user-authenticated interfaces from Google indexing.

Example 2: AI Scraper Controlled Policy

Input: Allow Googlebot and Bingbot; block GPTBot and CCBot from scraping training data.
Calculated Result: Granular multi-agent rules array specifying distinct allow/disallow paths per bot.

Protects proprietary content while maintaining organic search rankings.

Common use cases

Next.js App Router Architecture

Deploy native framework robots.txt files without maintaining static public/ text assets.

Admin & API Route Protection

Prevent search engines from wasting crawl budget on private application endpoints.

AI Model Crawler Governance

Control whether OpenAI, Anthropic, and Common Crawl bots ingest site content.

Environment-Specific Crawl Rules

Disallow all crawlers on Vercel preview/staging environments while allowing production.

Web Developer Tools

Next.js Metadata Generator

The Next.js Metadata Generator creates production-grade, type-safe Next.js App Router metadata configurations,...

Launch Tool →
Web Developer Tools

Next.js Sitemap Generator

The Next.js Sitemap Generator constructs idiomatic `app/sitemap.ts` code for the Next.js App Router, combining...

Launch Tool →
Technical SEO Tools

Robots.txt Tester

The Robots.txt Tester evaluates website crawling permissions according to the RFC 9309 Robots Exclusion Protoc...

Launch Tool →

Frequently asked questions

Where should robots.ts be located in Next.js?

Create the file at `src/app/robots.ts` (or `app/robots.ts`). Next.js will automatically compile and serve it at `/robots.txt`.

Can I block staging and preview deployments in Next.js?

Yes. You can inspect `process.env.VERCEL_ENV === 'preview'` inside `app/robots.ts` and return `disallow: ['/']` to prevent non-production indexing.

Does robots.txt prevent pages from appearing in Google?

Robots.txt prevents crawling, but if other websites link to the page, Google may still index the URL without content. Use `<meta name='robots' content='noindex'>` for guaranteed removal.

What is the official type for Next.js robots?

Import `MetadataRoute` from `'next'` and type the function return as `MetadataRoute.Robots`.

Blueprint Grid Background
AVAILABLE FOR NEW CONTRACTS & ARCHITECTURAL BUILDS

Let's build something
exceptional together

Work directly with Faisal Rafique to architect and deliver high-performance Next.js 15 platforms, 60fps Flutter mobile applications, and enterprise automated QA testing pipelines.

Start a Project Consultation→Explore Engineering Handbooks➔
✓Direct Senior Architect Access
✓100% Code & IP Ownership
✓Milestone-Based Global Delivery