Why your website is invisible to ChatGPT

Not every AI crawler does the same job. Some fetch your page to answer a question a person is asking right now — those are the ones that decide whether your business gets named. OpenAI's official crawler documentation explains:

“OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

Others only gather text to train future models. Blocking the first group removes you from AI answers; blocking the second is a reasonable business decision. Search crawlers and training crawlers are easy to confuse and are often blocked together by accident.

In our first audit round, 99% of the 353 pages had at least one critical issue, and 95% were flagged for no named, credentialed person behind the content, so even when an AI crawler can reach a page, there's often little worth quoting.

Google publishes its own crawler documentation distinguishing the crawlers used for search from those used for AI training, including Google-Extended.

Google's crawler waits for your page to finish loading. The AI crawlers do not.

Many modern websites deliver a mostly-empty page and then fill it in a split second later. Google handles this fine; it waits. But the crawlers behind ChatGPT, Claude and Perplexity read the page as it first arrives and move straight on. Anything that appears even a moment late is, to them, not there at all. Independent testing by Vercel and MERJ found AI crawlers request resources far less often than Googlebot and frequently do not execute JavaScript at all.

In our own first audit round — 353 pages across 19 websites — 19% of pages were missing main content from the raw HTML AI crawlers read. That's roughly one in five pages where content a visitor sees doesn't reach an AI crawler that skips JavaScript.

So a page can look perfect to you, rank respectably on Google, and still be a blank brochure as far as every AI assistant is concerned.

You don't need a paid tool to find out. Three quick checks:

• View source, not DevTools. Right-click the page and choose "View page source." That is the raw HTML the crawler receives. DevTools shows the page after JavaScript has run, which is not what an AI crawler sees.
• Disable JavaScript in your browser and reload. Anything that vanishes is invisible to AI crawlers that don't execute scripts.
• Search the source for your prices, service area or phone number. If they aren't in the raw HTML, they aren't there for an AI assistant.

Even when your robots.txt allows a crawler, a CDN rule, WAF policy or server-side bot filter can still return a 403 to it. Many security plugins and CDN default settings can treat unknown user-agents as hostile and block them silently. The result: your page allows AI crawlers on paper but refuses them in practice, and you'll never see an error in your analytics.

Check your CDN and firewall logs for requests from the search crawlers in the table above. If they're returning 403 or 1xxx codes, your page is blocked at the edge, not in robots.txt.

What an AI assistant actually needs on a page in order to name you: real prices, actual service areas or delivery times, your guarantee or returns policy, opening hours, a customer's own words. If the page is well-written but generic — the sort of thing any competitor could publish word for word — it offers nothing an AI assistant can uniquely cite, no matter how good everything else is.

Our audit data backs this up. Of the 353 pages we graded in our first round, 91% were flagged on the non-commodity test — they contained nothing a competitor couldn't also say — and 91% were flagged for few or no concrete numbers for an AI engine to quote. On average each page carried 40.2 issues, and 82% contained passages that don't make sense when lifted out on their own.

Promotional tone measurably reduces how often AI systems cite a page. This isn't a matter of taste. A page that says "we're the leading provider of world-class solutions" is less likely to be quoted than one that says "we install heat pumps in Sevier County, most jobs done in two days, from $4,200."

Research on generative engine optimization (Aggarwal et al., arXiv:2311.09735) found that citing sources, adding statistics and making content more quotation-friendly increased visibility in AI-generated answers by up to 40%.

What gets a page quoted:

• Specific numbers rather than vague claims
• Facts in real tables
• Sections that still make sense when quoted on their own
• Consistent naming so a machine knows your business is one business rather than four similarly-named ones

The findings below come from our first audit round: 353 pages across 19 websites. Counts include issues graded FAIL or WARN, and each issue is counted at most once per page. One website contributed 205 of the 353 pages, so its results weigh heavily in these figures.

Average: 40.2 issues per page. Counts include issues graded FAIL or WARN, and each issue is counted at most once per page. One website contributed 205 of the 353 pages, so its results weigh heavily in these shares.

See how those findings look in a finished report: the full, unredacted audit of our own homepage, with the grade, evidence and ranked fixes.

These fashionable tactics are ones we will not recommend, and the reasons are documented by the platforms themselves:

• Special "AI instruction files" (llms.txt) that Google has stated it does not use for ranking. As Google's John Mueller stated regarding llms.txt:

  “I don’t think anyone knows – it’s purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?).”

• A separate page for every search phrase
• Buying reviews or mentions
• FAQ and HowTo are two types of search markup that Google retired from rich results but which agencies still sell

You can't improve what you can't see. Three places to look:

• GA4 referral segment. Filter sessions by source to see traffic arriving from chatgpt.com, perplexity.ai, claude.ai and copilot.microsoft.com. The volumes are small today but growing.
• Bing Webmaster Tools AI Performance report. It shows how often Copilot and Bing's AI answers cite your pages, which pages are cited, and the "grounding queries" Copilot searched to find them. Newer views add citation share, topics and intents. It doesn't report clicks: citations measure visibility, not traffic, so pair it with the GA4 segment above. (Bing's announcement)
• Google Search Console. The Search generative AI performance report (rolled out worldwide in August 2026) shows impressions from AI Overviews and AI Mode by page. It doesn't show clicks or queries yet.

If the checks above turned up a problem, here are the fixes that cover almost every case. Each is something you can do yourself or hand to a developer in one sitting.

Copy-paste robots.txt for AI search crawlers

The safest default is to allow every search and retrieval crawler and to block only the training crawlers you've decided not to feed. This block does that, and it names each bot so a reader can see the intent. Add this to your /robots.txt:

# Allow AI search and retrieval crawlers (removing these = invisible in AI answers)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bingbot
User-agent: Googlebot
Allow: /

# Training crawlers — block only if you choose not to contribute training data
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Everyone else, plus any paths you already block. Copy each Disallow line into
# the Allow group above as well: named bots ignore this group.
User-agent: *
Allow: /
# Disallow: /cart

Sitemap: https://example.com/sitemap.xml

ChatGPT-User, Claude-User and Perplexity-User are live fetchers: they retrieve a page the moment a person asks a question. Claude-User obeys robots.txt, so blocking it there removes you from Claude's live answers. OpenAI and Perplexity say their live fetchers may ignore robots.txt because a person asked for the page, so for those two a CDN or firewall rule is what actually shuts them out. They're listed in the Allow group so your intent is clear. Bingbot is included because Bing powers Copilot's web results.

Test each bot with curl

To confirm a bot can fetch your page, test it from a terminal with that bot's user-agent. A 200 OK response means your server and any user-agent rules let that name through. It can't prove your CDN lets the real bot through, because CDNs identify verified crawlers by where the request comes from, not just the name. To confirm, look for real requests from each bot in your CDN's security or bot logs (in Cloudflare: Security Events or AI Crawl Control):

curl -A "OAI-SearchBot" -I https://example.com/your-page
curl -A "ChatGPT-User" -I https://example.com/your-page
curl -A "Claude-SearchBot" -I https://example.com/your-page
curl -A "Claude-User" -I https://example.com/your-page
curl -A "PerplexityBot" -I https://example.com/your-page
curl -A "Perplexity-User" -I https://example.com/your-page
curl -A "Bingbot" -I https://example.com/your-page

If those return 403 or a 1xxx code, your robots.txt is fine but your CDN or firewall is blocking the bot at the edge — see the Cloudflare fix below. Cloudflare's 1xxx codes are its own edge errors: 1010 and 1020 mean a bot or firewall rule refused the request. (A 1016 is different: your origin's DNS is broken for everyone, not just bots.)

Fix JavaScript-only content (SSR, prerendering, static generation)

If your prices, services or contact details vanish when you disable JavaScript, the fix is to get that content into the raw HTML the server sends. Three approaches, in order of how thoroughly they solve it:

• Server-side rendering (SSR). Render the page on the server so the HTML a crawler receives already contains the content. Frameworks like Next.js, TanStack Start, Nuxt and SvelteKit support this out of the box. This is the most complete fix — every crawler sees the same HTML a visitor does.
• Prerendering. Frameworks can prerender static routes at build time, and hosts like Netlify, Vercel and Cloudflare Pages serve the result. A prerendered page is a complete HTML file on disk, so no crawler has to run JavaScript.
• Static generation. If your content rarely changes, publish it as plain HTML. A static site has no JavaScript rendering gap at all — the raw HTML is the page.

A partial fix that often helps: move the critical text — your H1, your prices, your service area, your phone number — out of client-side components and into the server response, even if the rest of the page still hydrates. The audit's check #5 flags any page where more than 20% of main content is missing from the raw HTML. See every check the audit runs for the full list.

Unblock AI search bots in Cloudflare

Cloudflare's settings are a common reason a page allows AI crawlers in robots.txt but still returns a 403 or 1xxx code to them. Since July 1, 2026, Cloudflare sorts AI crawlers into three categories: Search (crawlers that index your pages to answer questions later, such as OAI-SearchBot and PerplexityBot), Agent (fetchers acting for a person right now, such as chat assistants' live fetches) and Training. It replaced the old single "Block AI bots" toggle. On every plan, including Free, you can set each category to allow, block, or block only on pages that show ads, under Security → Settings. See Cloudflare's change log and announcement.

Since September 15, 2026, domains newly added to Cloudflare block Training and Agent on pages with ads by default. If your pages carry ads, live fetches from ChatGPT or Perplexity can be refused even though Search is allowed and your robots.txt says yes. Set Search and Agent to Allow unless you have a reason not to.

Also check Security → WAF: a custom rule that blocks "unknown" user-agents will refuse AI crawlers whatever the category settings say.

To confirm a change worked, look for real requests from each bot in Security Events or AI Crawl Control over the next few days. A curl test from your own computer only proves the user-agent string isn't blocked (see above).

Getting found faster in Bing and Copilot (IndexNow)

Bing powers Copilot's web results, and Bing accepts IndexNow, a protocol that lets you tell Bing (and any other participating search engine) the moment a page is published or updated, instead of waiting for the next crawl. It's a single HTTP request:

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "example.com",
    "key": "YOUR_KEY",
    "keyLocation": "https://example.com/YOUR_KEY.txt",
    "urlList": [
      "https://example.com/your-page"
    ]
  }'

Generate any string as your key, host it at /YOUR_KEY.txt so Bing can verify you own the site, then call IndexNow whenever you publish or meaningfully update a page. Bing re-crawls the page on its own schedule. IndexNow doesn't affect Google (it has its own crawl scheduling), but it's the fastest way to tell Bing and Copilot a page has changed.