GEO
What AI search engines actually read on your site
ChatGPT and Perplexity do not browse your site the way a person does. Here is what they fetch, what they skip, and why a beautiful page can be invisible to all of them.
Leer másGEO
Is your site blocking ChatGPT without you knowing?
Two lines in a file most people never open can keep your business out of every answer ChatGPT gives. Here is how to check, and what to do about it.
Por AIVEYO4 minutos de lectura
If your robots.txt contains User-agent: GPTBot followed by Disallow: /, then ChatGPT cannot read your website. And a business ChatGPT cannot read is one it will almost never recommend.
Checking takes about thirty seconds. Open yourdomain.com/robots.txt in a browser and search the page for four names: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. This is what a block looks like.
robots.txtPlain text
User-agent: GPTBot
Disallow: /Why this matters now
A growing share of buying research now starts inside an AI assistant rather than on a results page. Someone asks ChatGPT for the best accountant in Manchester, or for alternatives to a product they already pay for. The assistant reads a handful of sources and names a handful of businesses. There is no page two.
If a crawler cannot fetch your pages, you are not in the pool it reads from, so you are not in the answer. What makes this hard to catch is that nothing looks wrong from where you are standing. Your Google rankings hold. Your traffic dashboard looks normal. You are simply absent from a channel that never reports back.
How this happens by accident
Almost nobody blocks AI crawlers deliberately and then forgets about it. The usual routes are quieter than that.
- Security and bot-protection plugins. Plenty of block bad bots tools shipped blanket lists that happened to include AI crawlers. One toggle, set years ago, still applying today.
- Agency boilerplate. Some agencies added block AI lines to every robots.txt they wrote in 2023 and 2024, back when the argument was purely about scraping. Sites built from those templates still carry it.
- CDN and hosting defaults. Some bot-management settings turn AI user agents away even when robots.txt is spotless. The crawler gets a 403 instead of your content, and your file looks innocent.
- A 2023 decision nobody revisited. Blocking training crawlers was a reasonable call at the time. Those same lines now decide whether you show up in AI search, and that trade has changed.
Know which crawler is which
| Crawler | Run by | What blocking it costs you |
|---|---|---|
| GPTBot | OpenAI | Training data, and some retrieval |
| OAI-SearchBot | OpenAI | Your place in ChatGPT search answers |
| ClaudeBot | Anthropic | What Claude knows about your site |
| PerplexityBot | Perplexity | Your listing in an AI search engine |
But I do not want AI companies taking my content
That is a fair position, and a genuine trade-off rather than a myth. If your content is the product, paywalled journalism or research you sell, restricting access can be exactly the right call.
For most businesses it is not the right call. The website exists to be found, and the content on it is marketing. Blocking crawlers to protect marketing copy is a bit like going ex-directory so competitors cannot look you up. It works, and it costs you more than it saves.
The point is that it should be a decision, not an accident.
There is a middle path, and it is the one most businesses actually want. Allow the retrieval crawlers so you can be cited, block the training ones if that matters to you, and write your preferences down in an llms.txt file.
How to check, and how to fix it
- Read your robots.txt. Open yourdomain.com/robots.txt and search for each of the four names. A block is the crawler name followed by a disallow line.
- Look past robots.txt. A clean file does not guarantee access, because CDN bot protection can still turn crawlers away. Our free AI Crawler Check tests real access for all four at once.
- Remove or narrow the block. Delete the disallow lines, or scope them. Keeping a private folder closed while opening the rest is perfectly reasonable.
- Check the layer above. If the block lives in a plugin or your CDN rather than the file, find the bot protection setting and allow the four crawlers there instead.
- Verify after you deploy. Re-run the check. Changes are usually picked up within a few hours, occasionally a few days.
If you want to stay out of training data while still being citable, this is the shape to aim for.
robots.txtPlain text
# Not for model training.
User-agent: GPTBot
Disallow: /
# Yes to being cited in answers.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /Access is the front door, not the house
Letting crawlers in is necessary, and it is not enough on its own. Once inside, an engine still has to pull a clear answer out of your page, trust it, and prefer it to whatever a competitor published. Crawler access is the first of six signal groups we score.
The free scan shows all six for your site, engine by engine, in about ten seconds. No signup, and you get a link you can share with whoever needs to see it.