Is your site blocking the AI crawlers that could cite you?

In 2023 and 2024 a lot of organisations added directives to block AI crawlers. At the time it was a reasonable position: nobody wanted their content used to train commercial models without compensation, and the block was cheap to apply.

Then the ground shifted. The same crawlers that read content for training also, in many cases, read content for live retrieval, the process that decides which sources an AI assistant cites when answering a question today. A directive added to protect intellectual property is now, for a lot of companies, quietly removing them from the answers their buyers see.

Most teams have not revisited the decision. Some do not know it was made.

Check what you are currently allowing

Open yourdomain.com/robots.txt and look for these user agents:

| Agent | Operated by | Primarily controls | |---|---|---| | GPTBot | OpenAI | Crawling for training | | OAI-SearchBot | OpenAI | Search and retrieval | | ChatGPT-User | OpenAI | Live fetches during a user's session | | ClaudeBot | Anthropic | Crawling | | PerplexityBot | Perplexity | Search and retrieval | | Google-Extended | Google | AI training use, not Search indexing | | Bingbot | Microsoft | Search indexing, which feeds several assistants |

The specific agents and what each controls change over time. Verify against each provider's current documentation rather than trusting a list, including this one.

Two things people commonly get wrong:

Google-Extended does not control AI Overviews. It governs whether your content is used for training certain Google models. AI Overviews draw on the standard Search index. Blocking Google-Extended therefore does not remove you from AI Overviews, and does not protect you from the traffic effect people usually think it does.

Blocking Bingbot has knock-on effects. Several assistants build their candidate set from search indexes. Restricting conventional crawling can remove you from retrieval on surfaces you never explicitly configured.

The decision is not one decision

Conflating training and retrieval is where most of the confusion sits. They are separate choices with separate consequences.

Blocking training means your content is not used to teach future models. The effect on whether you are cited today is close to zero, because current models are already trained. The effect on whether a future model knows who you are is real but distant and unmeasurable.

Blocking retrieval means that when someone asks an assistant a question your content could answer, your page is not in the candidate set. You are not cited. No link, no mention, no attribution. That effect is immediate and it applies to every relevant query.

For most service businesses these should be different answers. There is a coherent position that says: do not use my archive to train your commercial model, but do read my site when a potential customer is asking a question I can answer.

Who has a genuine case for blocking

Publishers whose revenue depends on page views. If an AI answer satisfies the query without the visit, the citation does not pay for itself. That is a real economic conflict and the blocking decision follows from it.

Sites with content that genuinely should not be reproduced: proprietary research behind a paywall, licensed material, anything with contractual restrictions.

Almost everyone else is blocking their own distribution. If you sell services and your problem is that not enough of the right people know you exist, being excluded from the systems increasingly used to find providers is not protection. It is absence.

Blocking is not the only way to be invisible

Robots directives are the explicit version. There are quieter failures that produce the same outcome, and in my experience they are more common:

Content that requires JavaScript to appear. If the substance is not in the initial HTML response, assume it will not be read. Some crawlers execute JavaScript, many do not, and you cannot control which one arrives.

Content behind an interaction. Accordions, tabs, and read-more toggles that load content on click rather than hiding already-present markup.

Slow or unreliable responses. A fetch that times out is a fetch that did not happen.

Aggressive bot protection. WAF rules and rate limits configured against scrapers will often catch legitimate AI crawlers alongside them. This one is worth checking specifically, because it is usually configured by a different team than the one responsible for marketing visibility, and nobody has connected the two decisions.

Geographic or consent gates that serve an interstitial before content.

What to do about it

Audit what you currently allow. Read your own robots.txt properly. If you cannot explain why each directive is there, that is the finding.

Decide training and retrieval separately. Write down the reasoning. This decision gets revisited by someone else in two years and they will need to know what you were solving for.

Verify rather than assume. Fetch your key pages as a crawler would and confirm the content is present in the raw HTML. Check your server logs for AI user agents: whether they are arriving at all, and what response codes they are receiving. A 403 rate you did not know about is a common and completely fixable discovery.

Check the bot protection layer, not just robots.txt. The block is often there rather than in the file everyone looks at.

On llms.txt

You will see this recommended. It is a proposed convention: a plain-text file at your root summarising your site's important content in a form easy for a language model to consume.

Honest assessment: it is not an official standard, adoption by the major AI systems is limited, and there is little public evidence of it affecting citation. It is cheap to add and unlikely to do harm. Add it if you like, but not before your robots.txt is correct and your content renders server-side. Those two things determine whether you are readable at all.

The check worth running first

Before optimising anything, confirm you are not excluded. Fetch a key page as an AI crawler would see it. Read your robots.txt line by line. Check your logs for what those user agents actually receive.

It is the least interesting work in this whole area and it is the layer everything else depends on. A site that cannot be reached does not benefit from better content, cleaner structure, or more consistent entity signals. None of it applies to a page that was never read.

This article is part of the SEO in the AI Era: The Complete Guide guide.