Can AI crawlers read a Next.js app?

Yes, as long as the text is rendered on the server. Server components, static generation and SSR all put content in the initial HTML. Content fetched in the browser after load is invisible to most AI crawlers.

Should llms.txt go in public/ or a route handler?

public/llms.txt is simplest and fine for small sites. A route handler lets you proxy an auto-updating hosted copy or build the file from your CMS at request time.

Does Vercel block AI crawlers?

Not by default. If you turn on firewall or bot-protection rules, check that the AI crawlers you want are still allowed — the audit tests your live site.

Will an llms.txt get my docs cited?

It helps agents find and read the right pages, but no file can guarantee citations. Crawler access, server-rendered content and clear structure matter at least as much.

Next.js · AI search readiness

Make your Next.js site AI search ready

Next.js gives you full control over robots, metadata and rendering — which also means every gap is yours to fix. Check what AI crawlers actually receive, then wire up robots.ts, llms.txt and structured data in minutes.

Free · live crawl · includes a generated llms.txt

What Next.js gets right

Server components and static generation put real text in the initial HTML, which is what AI crawlers read.
app/robots.ts and app/sitemap.ts generate both files from code, so they stay in sync with your routes.
The Metadata API makes per-page titles, descriptions and Open Graph tags straightforward.

Where Next.js sites lose AI visibility

Client-only pages

Pages marked 'use client' that fetch their content in useEffect ship an empty shell. AI crawlers don't run JavaScript, so they see nothing. Fetch on the server instead.

Preview deployments getting crawled

Indexable preview URLs can be quoted instead of your domain. Keep previews behind deployment protection or return a noindex robots.txt outside production.

Bot protection that blocks AI crawlers

Firewall or bot-management rules that challenge unknown user-agents also block GPTBot and PerplexityBot. Allow the AI crawlers you want explicitly.

No structured data

Next.js doesn't add Schema.org markup for you. Render Organization JSON-LD in the root layout and Article or Product JSON-LD on detail pages.

Real data · measured October 2026

vercel.com itself scores 81/100 B-

4 of 4 core AI crawlers allowed · publishes an llms.txt · median across the 102 sites we measured is 77. Your Next.js site starts from the same platform, but your settings and content decide the score.

See the breakdown

Step by step

Make Next.js AI-ready

Allow AI crawlers in app/robots.ts

Generate robots.txt from code so the rules ship with every deploy.

app/robots.ts

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/" },
      { userAgent: ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "PerplexityBot", "ClaudeBot", "Claude-SearchBot", "Google-Extended"], allow: "/" },
    ],
    sitemap: "https://yourapp.com/sitemap.xml",
  };
}

2
Generate your llms.txt
Run the audit above. LLMready reads your sitemap and navigation and writes llms.txt and llms-full.txt you can drop straight into your repo.

Serve it at /llms.txt

Put the file in public/llms.txt for a static copy. To keep it current without redeploys, proxy your hosted LLMready file from a route handler instead (Pro).

app/llms.txt/route.ts

export const revalidate = 3600;

export async function GET() {
  const res = await fetch("https://answer-engine-ready.lovable.app/h/your-site/llms.txt", { next: { revalidate: 3600 } });
  return new Response(await res.text(), {
    headers: { "content-type": "text/plain; charset=utf-8" },
  });
}

Add Organization JSON-LD

Render it once in the root layout so every page tells answer engines who you are.

app/layout.tsx

const org = {
  "@context": "https://schema.org",
  "@type": "Organization",
  name: "Your company",
  url: "https://yourapp.com",
  logo: "https://yourapp.com/logo.png",
  sameAs: ["https://www.linkedin.com/company/yourcompany"],
};

// inside <body>
<script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(org) }} />

Next.js AI search FAQ

See your Next.js site's AI readiness score

Shopify AI search guide WordPress llms.txt guide Webflow AI indexing guide Framer AI search guide Wix AI search guide Squarespace AI search guide Ghost llms.txt guide llms.txt generator Industry benchmarks