AI Crawlability
AI Crawlability for B2B Companies
Answer engines cannot cite what they cannot fetch. We work through the access, content and entity layers that decide whether your most important pages are readable by each system — and where the rules quietly contradict each other.
User-agent: *Allow: / User-agent: GPTBotAllow: / User-agent: ChatGPT-UserAllow: / User-agent: ClaudeBotAllow: / User-agent: Claude-UserAllow: / User-agent: PerplexityBotAllow: / User-agent: GooglebotAllow: /Excerpt from this site's live file — /robots.txt
Access surface
Crawler access matrix
Every AI vendor reads its own user-agent name. A site can therefore be open to one system and invisible to another without a single error appearing anywhere.
Illustrative interface. Rows show the agents named in this site's own public robots.txt and how such a file is read — they are not the result of a check performed on your website.
| Crawler / system | Public access signal | Status example | Why it matters |
|---|---|---|---|
| GPTBotOpenAI training crawler | robots.txt user-agent group | Available | A published directive is the only public statement about whether this agent may read the site. |
| ChatGPT-UserOpenAI on-demand fetch | robots.txt user-agent group | Available | Assistant browsing uses a separate agent, so a rule written for the training crawler does not cover it. |
| ClaudeBotAnthropic crawler | robots.txt user-agent group | Available | Each vendor reads its own agent name, so omissions are silent rather than reported. |
| Claude-UserAnthropic on-demand fetch | robots.txt user-agent group | Review required | Paired on-demand agents are easy to omit, and when one is missing access falls back to the wildcard group rather than an explicit rule. |
| PerplexityBotPerplexity crawler | robots.txt user-agent group | Available | Answer engines that link sources need retrievable pages before a citation is possible at all. |
| GooglebotGoogle Search crawler | robots.txt user-agent group | Available | Search indexing and AI surfaces share the same retrieval layer, so a block here affects both. |
| Google-ExtendedGoogle generative use control | robots.txt user-agent group | Restricted | This token controls generative reuse only — restricting it does not remove a page from ordinary search. |
- Available
GPTBot
OpenAI training crawler
- Available
ChatGPT-User
OpenAI on-demand fetch
- Available
ClaudeBot
Anthropic crawler
- Review required
Claude-User
Anthropic on-demand fetch
- Available
PerplexityBot
Perplexity crawler
- Available
Googlebot
Google Search crawler
- Restricted
Google-Extended
Google generative use control
Three layers
Being reachable is only the first of three problems
Teams usually fix the first layer and assume the job is done. Each layer can pass while the next one fails, and none of them reports an error.
- Access layer
- Directives file, sitemap, HTTP responses and canonical URLs
- Decides whether a system is allowed to look at all
- Content layer
- Server-rendered HTML, plain-text resources, internal links
- Decides whether anything readable is there once it does
- Entity layer
- Structured data, consistent naming, authoritative profiles
- Decides whether the reader can tell who the page is about
How we work
What a crawlability review looks like
A high-level view of the sequence. The detail depends on the stack, the CDN and how the site is deployed.
Read the public access signals
We start from what any visitor can fetch: the directives file, the sitemap and how key pages respond.
Compare rules against intent
Rules written years apart often contradict each other. We surface where the file says something the team did not mean.
Check what survives without JavaScript
Access is not enough if the answer to a buyer's question only exists after a client-side render.
Re-check after the site changes
A deploy, a CDN rule or a new subdomain can quietly reverse a fix, so the review is repeated rather than filed away.
What you get
Findings a developer can act on
The output is not a list of warnings copied from a tool. It is a set of observations tied to specific URLs, ordered by what blocks retrieval first, with the reasoning attached so your team can judge each one.
- Access observations with the affected URLs
- Where the public directives disagree with the team's intent
- Content that exists only after a client-side render
- A recheck point for after the next deploy
FAQ
Frequently asked questions
Next step
Ready to make your site citable by AI answer engines?
Tell us about your company and website. We will review the request and contact you to discuss the most relevant GEO and AI Visibility audit scope.