Complete Reference On Allowing & Blocking LLM Crawlers

Even as everyone wants to know how to be visible in AI search results and LLM sessions, a lot of sites are also blocking LLM crawlers. In some cases, these blocks are only for certain use cases: don’t let LLMs use the content to train the models or don’t let the information be used in Google’s AI overviews. In other cases, crawlers may be unknowingly blocked.

I dug into all the AI crawlers, why they might be crawling your site, and how to make sure your site is configured the way you intend for the results you want. I looked at how blocking configurations can impact retrieval-augmented generation and grounding/fan-out queries and how to opt out of Google’s AI overviews. And I compiled a list of all the major user-agents and what they’re used for. (Note that I didn’t look into bots used for ads; I focused solely on organic crawling.) And no, AI didn’t write this for me although it did help me compile some of the data and do some verification!

While there’s a lot that’s novel about the rise of LLMs, it’s also true that everything old is new again. In the earlier days of Google, some site owners felt Google shouldn’t be able to use their content (to power the search index) for free. For a time, some review sites blocked Google from using their reviews on Maps (this was actually a very complex situation and I am simplifying a lot). News publishers sued Google for linking to news stories. Publishers wanted Google to pay them for inclusion. Some publishers wanted to block Google’s access to Google News, thinking this would cause readers to come to the sites directly. (Spoiler alert: this is not what readers did.) Organizations were formed to create complex protocols and rules that ultimately weren’t adopted.

In some cases, site owners ultimately decided it wasn’t worth the loss of audience to opt out. In other cases, Google has put together partnerships or created options for granular exclusion lists. And sometimes, Google properties just shuttered in certain countries. We’re seeing these discussions about stolen content and reduced clicks play out again and the best we can do for now is keep evolving to meet people where they are.

What Does It Mean to Block an LLM Bot: Training Data, Web Search, AI Answers, and User-Triggered Lookup

Look, before we get into all of this, it’s complicated and it’s changing all the time. And unlike the Google-blocking discussions of days past, the conversations today about blocking or not go beyond just whether or not a web corpus (like a search engine or LLM) should get to use content from other companies for free. LLMs can also be used to build databases and sites and apps on top of another company’s content. (The LLM effectively provides the connection to the content and builds the code!)

In many cases, there are certainly business cases both for and against blocking LLM access to site content. The best we can do right now is know the implications and the mechanisms.

At the highest level, there are three types of access (and corresponding blocking ability):

• corpus-time access – this includes training models and crawling pages for web search indexing. (Note that blocking search engines and LLMs from using your content in training data will only prevent its use in future training; all content it’s already trained on is baked into the models. Blocking your content from being indexed in web search means that as the index is refreshed, your content will no longer be included.)
• answer-time use – when an AI search feature (Google’s AI Overviews and AI Mode, Copilot, Siri, Gemini) builds an answer from pages already in its search index. Nothing crawls your site at that moment, so you control this with settings and meta tags rather than by blocking crawlers.
• user-triggered access – this includes webpage requests LLMs make during user sessions, agentic transactions, and MCPs.

What Each Setting Controls

Each way of blocking affects some of these uses and not others.

Setting	Training	Web search	AI answers	User-triggered lookup
robots.txt: disallow a training crawler (GPTBot, ClaudeBot, CCBot, Amazonbot, Meta-ExternalAgent)	Stops future training by that company	No effect	No effect	No effect
robots.txt: disallow Google-Extended	Stops Gemini training on your content	No effect	Removes your pages from grounding in the Gemini app and Vertex AI. No effect on AI Overviews or AI Mode.	No effect
robots.txt: disallow Applebot-Extended	Stops Apple foundation model training	No effect	No effect (Apple’s AI answers are controlled with nosnippet)	No effect
robots.txt: disallow an LLM search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot)	No effect	Keeps your pages out of that company’s search index	That assistant can’t find your pages when it searches the web	No effect
robots.txt: disallow Googlebot, bingbot, or Applebot	Google, Microsoft, or Apple can’t train on pages they can’t crawl	Out of that search engine (the URL can still appear, without its content)	Out of that company’s AI answers. Blocking Googlebot also blocks Brave, and so Claude’s web search.	No effect
robots.txt: disallow a user-triggered fetcher	No effect	No effect	The assistant may cite your page from its index but can’t open it	Works only for Claude-User and DuckAssistBot; the others ignore robots.txt
403 at the server or CDN, by user agent	Same as the matching robots.txt rule, but enforced. Doesn’t work for Google-Extended or Applebot-Extended, which never make requests.	Same as the matching robots.txt rule, but enforced	Same as the matching robots.txt rule, but enforced	Blocks the user-triggered fetchers that ignore robots.txt
noindex meta tag	Bing: also opts the page out of Microsoft’s AI training	Removes the page from the index	Removes the page from AI answers built from that index	No effect
nosnippet meta tag	No effect	Google: the page still ranks but shows no text snippet	Google: out of AI Overviews and AI Mode (not the Gemini app). Apple: out of AI answers in Siri and search.	No effect
noarchive meta tag	Opts the page out of Microsoft’s AI training (Amazon honors it as a training opt-out too)	No effect on Bing’s index	Keeps the page out of Copilot answers	No effect
Search Console > Search generative AI > Exclude	No effect (that’s Google-Extended)	No effect	Out of AI Overviews, AI Mode, and Discover’s AI features	No effect
Login or paywall shown to bots	Blocked	Blocked	Blocked	Blocked

Cloudflare’s settings map onto these rows as noted below. For more information specifically about Google:

• What each user-agent impacts
• What impacts AI features

Implications of Blocking LLM Bots

Keeping all of this straight is getting trickier. Here’s what I’ve determined.

Blocking LLM Crawling for Training Data

I’ve talked to several companies who want their site content to be available in web search results (in both Google and LLMs) but don’t want their sites used in training data. The logic seems to be that training data just gives the LLMs their hard-earned content for free with nothing in return. And that may be true for some educational/informational sites (LLMs often don’t provide citations when providing general knowledge). Across the sites Cloudflare serves, less than 1% block search bots, while 17% use some mechanism to block AI training.

Training data is how LLMs know about your brand and your site. The stronger your brand reputation and the more closely what you have to offer aligns with what the LLM user is looking for, the more likely you are to get visibility. This training data not only helps determine if LLMs mention you in user sessions but also influences the fan-out queries behind the scenes (see “what is RAG” below) and whether they look up information on your site directly or via web search. (I understand the situation is more complicated for publishers in particular, but when looking through the lens of LLM visibility, it helps to allow crawling for training data.) LLMs can also learn about product details, specs, press, and reviews.

So my recommendation in most cases, honestly, is to let the LLMs train on your data.

In addition, blocking training can have additional implications. Blocking Google-Extended also takes your content out of Gemini’s grounding and Cloudflare’s Block setting for training also blocks Googlebot.

Blocking LLM Crawlers for Web Search Data

If you allow Google and other traditional search engines to index your site, you probably want to allow LLM crawlers to index your site as well. Blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot reduces or removes your visibility when ChatGPT, Claude, and Perplexity search the web to answer a question.

Initially, LLMs seemed to rely on traditional search engines for their web search (ChatGPT may use Bing, Claude uses Brave, everyone may somewhat secretly use Google), but not surprisingly, they are also building their own search indices. Site owners are more inclined to allow this access as it’s what enables those sites to show up when the LLM does a web search (and most of the time, that includes a citation and link). Blocking Googlebot also blocks Brave, which won’t crawl what Googlebot can’t.

Blocking LLM Crawlers for User-Triggered/Agent Lookups

For an ecommerce site, this is the result of blocking these crawlers. Claude couldn’t access the page so couldn’t provide comparison or purchase information.

How to Block

You can block crawlers for training, web search, and user-triggered requests, and opt out of AI answers, in a number of ways.

• robots.txt – many LLMs and traditional search engines respect robots.txt directives aimed at training or web search user agents. (Note that most user-triggered fetchers ignore robots.txt directives as they treat them as requests from the user who triggered them. Anthropic’s Claude-User and DuckDuckGo’s DuckAssistBot are the exceptions.)
• robots meta noarchive/nosnippet tag – Bing/Copilot doesn’t have a user agent available to block specifically for training data but you can add a noarchive meta tag to any pages you don’t want used in AI training (it also keeps them out of Copilot answers). For Google and Apple, the nosnippet meta tag keeps a page out of AI-generated answers (see Opting Out of AI Answers below).
• WAF or CDN rules applied to published IP lists. (The IP lists are linked in Verifying User Agents below.)
• htaccess – you can configure an .htaccess file to return a 403 forbidden response based on user agent. This method only works for user agents presented during crawling. (For instance, Google crawls sites using Googlebot but then doesn’t use the output for AI training if you’ve blocked Google-Extended with robots.txt. But since Google doesn’t actually crawl with Google-Extended, configuring the server to return a 403 for this user agent won’t have any impact.)

Using robots.txt

LLMs like Claude and ChatGPT use specific user-agents for crawling sites for training so you can block those user-agents directly using the robots.txt exclusion protocol.

Google and Apple don’t crawl sites for training data separately from their regular web crawls so instead you can indicate whether that regular web crawling can be used for training by blocking the Google-Extended and Applebot-Extended user agents respectively. This means that even though there’s only one crawl (and the user agent used during that crawl doesn’t include “Extended” in the name), you can block usage for only training data and not web search indexing. (Google-Extended also controls whether Gemini can use your content for grounding, so blocking it can cost you Gemini visibility.)

If you want to block LLMs from using your content for training data, you may want to block Common Crawl as well (CCBot). Common Crawl is a non-profit that maintains a corpus of web crawl data for anyone to use (founded in 2007, with archives back to 2008). LLMs use Common Crawl data extensively for training.

You don’t block access using a new LLM-specific directive; it’s the same one used to block traditional web search user agent (Disallow). You can set a disallow directive for specific site sections or the entire site. See the LLM Bot Usage Summary below for a list of crawlers.

An example:

User-agent: ClaudeBot
Disallow: /folder-of-great-stuff/

Note that a crawler follows only the most specific group that names it, so once you add a group for ClaudeBot, ClaudeBot ignores everything in your User-agent: * group. Repeat any site-wide rules you still want it to follow.

A common setup is to block AI training but leave search and user-triggered crawlers open:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: MistralAI-Training
Disallow: /

Microsoft doesn’t have a training token, so add a noarchive meta tag for Bing (see Opting Out of AI Answers below).

Requiring a Login

If you require users to log in (with either a free or paid account) and you serve that same login to bots, then the bots won’t be able to access the content.

Blocking at a Web Application Firewall (WAF) or CDN

Use a CDN like Cloudflare or Akamai to detect the user agent and return a 403 (forbidden) or 402 (that points at licensing). This can also be URL pattern or page specific. For example:

(http.user_agent contains "ClaudeBot") and (http.request.uri.path contains "/folder-of-great-stuff/")

Blocking Web Search Indexing

A full rundown of blocking or allowing access of traditional search engines for web search indexing is beyond this scope but a few notes:

• Blocking via robots.txt prevents crawling but technically doesn’t prevent indexing. Google may learn about the URL from external links for instance, so may index the URL itself (without its content) and meta data based on those external links and other third party input. In practice, this tends not to be a problem as there’s little the URL would rank for when Google doesn’t know anything about the page content (historically Google set this up because, for example, back in the day large brands would accidentally block their home page and so if Google knew that home page URL existed, they would still want to rank it for brand searches). I generally recommend just using robots.txt in most cases since you get the crawl efficiency benefit. But technically, to keep a page from being indexed at all and to remove even the possibility it would show up in search results, use the noindex meta data instead.
• Excluding a URL from an XML Sitemap doesn’t prevent it from being crawled and indexed. (Just like including a URL doesn’t guarantee it will be crawled and indexed.) XML Sitemaps are one among many discovery mechanisms.
• Using the Google Search Console URL removal tool isn’t a permanent solution. It’s intended to quickly remove a URL (faster than a recrawl/deindex process would take). You’ll have to also put in a permanent fix such as a noindex, robots.txt disallow, 404/410, or login. Otherwise, the URL will show up in search results again in about six months. Also, the URL removal tool functionality is entirely related to the search results display. It doesn’t impact the actual web search index so if you’re trying to remove a bunch of low-quality or spammy pages from your site so that Google views the overall site as higher quality, using the URL removal tool on those pages won’t help. (Also don’t use this tool to remove non-canonical URLs from Google’s index, as that can cause the canonical URL to be removed as well, based on how Google clusters non-canonical and canonical URLs together in its index.)

Blocking User-Triggered Requests

Most user-triggered fetchers ignore robots.txt, because the vendors treat the request as coming from the user rather than from a crawler. The exceptions are Anthropic’s Claude-User and DuckDuckGo’s DuckAssistBot. To block the others, return a 403 based on the user agent at the server or CDN (see the WAF example above). Block these by user agent rather than IP address: some of them run on the user’s own computer (Claude Code, for instance), so their requests come from ordinary home and office networks.

Opting Out of AI Answers

You can specifically opt-out of AI results (AI overviews, AI mode, AI features in Discover) in Google Search Console (Settings > Search generative AI).

Note that opting out might also keep your site from modules like top stories, as that is increasingly being powered by AI overviews. So while opting out of AI results won’t impact traditional web search visibility, more and more search results components are shifting to being sourced from AI results rather than web search results so the impact may be greater than you expect. (Note that this Google Search Console control doesn’t impact use of a site’s content in AI training models.)

Google Search Console does provide impression-level data by URL on AI overviews so you can see what visibility you may be losing by opting out (Performance > Search Results > Generative AI).

You can also opt out of Google’s AI overviews and AI mode by adding the nosnippet meta tag to your pages. Note that this also removes the snippet from the organic search listings (the page still ranks but displays only the URL and title). This doesn’t impact Gemini visibility. You can also opt out of Apple’s AI answers in Siri and search with the nosnippet meta tag. For Copilot, use the noarchive meta tag, which also opts the page out of Microsoft’s AI training.

And note that you can also see Copilot visibility in Bing webmaster tools (and a report of grounding queries as well).

LLM Bot Usage Summary

Many of the crawler lists I encountered when researching this article were out of date, so this is the list I’ve compiled that as far as I can tell is accurate as of October 2026. No promises for the future though. I’m maintaining a tools page that includes a crawler check to determine what your site is currently allowing and blocking, as well as a Google Sheet with the details below (and more). I do have a Claude process set up to review and update the Google Sheet every week so it stays up to date.

LLM/Search Engine	Training	Web Search	User-Triggered Access	Notes
OpenAI/ChatGPT	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic/Claude	ClaudeBot	Claude-SearchBot	Claude-User	There’s no need to specify anthropic-ai in your robots.txt file as Anthropic no longer documents it (if you set up blocking with this user agent, you’ll need to update your robots.txt file to the latest user agent strings). Also note that the IP list json file does not cover requests from the MCP connector or Claude Code. See the documentation on the MCP connector and API tool requests.
Common Crawl	CCBot			Open web corpus used by LLMs for training models
Google	Google-Extended	Googlebot	Google-Agent FeedFetcher-Google GoogleMessages Google-Pinpoint GoogleProducer Google-CWS Google-Read-Aloud Google-Site-Verification Google-GeminiNotebook	Block Googlebot to block all Google web search indexing user agents; use more specific user agents to block for specific purposes (such as Googlebot-Image) You can’t block user-triggered requests with robots.txt as those user agents ignore robots.txt. To block them, return a 403 based on user agent at the server or CDN.
Google Vertex/Agent Search			Google-CloudVertexBot	Agent Search enables you to create AI-powered search from site content. If you don’t want your site’s content used in an Agent Search app, you can block access. (Note that blocking Googlebot also blocks this crawler.) (Cloudflare classifies this user agent as training so if you block AI training crawlers in Cloudflare, this will be blocked as well.)
Apple	Applebot-Extended	Applebot		You can also use the meta robots nosnippet tag to keep content from AI-generated answers.
Yahoo		Slurp		Yahoo primarily uses Bing’s web search index but does supplementary crawling with Slurp for its mobile search results and partner sites.
Meta/Facebook/Instagram	Meta-ExternalAgent	Meta-WebIndexer	Meta-ExternalFetcher FacebookExternalHit	FacebookExternalHit is used for link previews in social sharing. Meta-ExternalAgent is used for training and may also be used to index content directly.
Amazon	Amazonbot	Amzn-SearchBot	Amzn-User	Amazonbot is used “to improve our products and services” and “may be used to train Amazon AI models.” Amazon says Amzn-SearchBot and Amzn-User don’t crawl content for AI training. Amazonbot also treats the noarchive meta tag as a training opt-out.
Amazon Bedrock			bedrockbot amazon-bedrock-knowledgebase-on-behalf-of-<hash>	This is the crawler used when an AWS customer connects a web crawler data source to a Bedrock knowledge base.
ByteDance/TikTok	Bytespider		TikTokSpider	ByteDance doesn’t publish documentation for either user agent.
Microsoft		bingbot		Indexes for Bing, Yahoo, DuckDuckGo and Copilot, and may supply part of ChatGPT search (although that is not documented). To opt out of Copilot training and answers, use meta robots noarchive. (This doesn’t impact the bing traditional search index.) If you opt out of web search with noindex, you are opted out of training as well. There appears to be no way to opt out of training only yet. Microsoft is building robots.txt support for a no-training preference, targeted for early 2027.
Perplexity		PerplexityBot	Perplexity-User	See the IP list json
Brave				Brave follows settings for Googlebot, so however you block or allow Google will also apply to Brave. (This means that if you block Googlebot, you may also remove the site from Claude’s web search.)
DuckDuckGo		DuckDuckBot	DuckAssistBot	DuckAssist provides AI-generated answers. DuckDuckGo primarily uses Bing for its web search index, but does supplementary crawling with DuckDuckBot.
DeepSeek				DeepSeekBot is often referenced as being used for both training models and web search but there’s no official documentation or evidence of this.
Yandex		YandexBot		Note that User-agent: Yandex matches every Yandex robot, including its ad robots, so use YandexBot to target search crawling. Block YandexAdditional and YandexAdditionalBot if you don’t want a site to appear in AI answers.
Mistral	MistralAI-Training	MistralAI-Index	MistralAI-User
xAI				Bot directories list several names for xAI’s crawler (xAI-SearchBot, GrokBot, xAI-Grok), and xAI doesn’t document any of them. There’s no official documentation on robots.txt support, user agents, or IP ranges.

Verifying User Agents

Back in the day, scrapers would spoof the Googlebot user agent in order to crawl sites and we at Google had to add a way to verify it. Since everything old is new again, the LLMs are now making IP ranges or reverse DNS patterns available.

When giving bots special treatment, you want to make sure they are who they say they are. A crawler like GPTBot or PerplexityBot coming from an IP address that isn’t on the vendor’s list is very likely fake. The clearest signs of a fake: one IP address cycling through several companies’ bot names or requests for paths like /.env. But don’t treat a missing IP match as proof of a fake for agents that run on the user’s own computer (Claude Code, for instance).

• OpenAI/ChatGPT
  • Training crawler IP addresses
    Web search crawler IP addresses
    User-triggered access IP addresses
• Anthropic/Claude
  • Anthropic crawler IP addresses (covers Anthropic’s crawlers and the Claude-User fetcher, but not the MCP connector or API tool requests, and Claude Code runs on the user’s own computer)
• Perplexity
  • PerplexityBot IP addresses
    Perplexity-User IP addresses
• Google
  • How to verify Google user agents
• Amazon
  • Amazonbot IP addresses
  • Amzn-SearchBot IP addresses
  • Amzn-User IP addresses
• Apple
  • Applebot IP addresses
• Microsoft
  • Bingbot IP addresses
• Common Crawl
  • CCBot IP addresses
• Mistral
  • MistralAI-User IP addresses
  • MistralAI-Index IP addresses

Meta, ByteDance, and xAI don’t publish IP lists, and Yandex offers reverse DNS lookups only.

A Note About Cloudflare

Cloudflare, which is used by more than 20% of websites, provides controls to block or allow search engine and LLM crawlers for training, web search, and user-triggered requests (which they call “agent”). On September 15, 2026, they said that new sites will be set up with these defaults:

• Sites without ads: allow all crawling (training, web search indexing, user-triggered lookup)
• Sites with ads: disallow crawling for AI training, allow crawling for web search indexing, block user-triggered lookup on pages that serve ads

This means that by default, newly onboarded ad-supported sites will be set to disallow training crawlers and to block agent requests of pages that serve ads.

(Note that this is different than what they announced in July 2026, when they said they’d block “multi-purpose” crawlers such as Googlebot, Applebot, and Bingbot for sites that blocked AI training.)

Cloudflare now calls these crawlers “mixed use.” In September, Cloudflare revised its approach and introduced Disallow AI Training, after talks with Google, Apple, and Microsoft. Cloudflare says Apple, Google, and Microsoft either honor the preference now or have made time-bound commitments to do so.

You can find your site’s Cloudflare configuration in Security > Settings > Configure AI bot policies.

You can set controls to block or allow search engine and LLM crawlers from training, web search indexing, and user-triggered lookup.

Because Cloudflare considers Googlebot, Applebot, and Bingbot to be “mixed use crawlers”, one of the settings seems super confusing (to me anyway!).

All three intents have the following options:

• Allow
• Block on pages with ads
• Block

That seems pretty straightforward.

But the training configuration has a fourth option: disallow AI training. On the surface, that doesn’t make a lot of sense. What’s the difference between training > block and training > disallow AI training?

• disallow AI training – blocks specific training crawlers (GPTBot, ClaudeBot, etc.) and blocks mixed-use crawlers from AI training using Google-Extended and Applebot-Extended in robots.txt (see the note about Bingbot below, as it’s not controlled by this setting)
• block – blocks specific training crawlers and mixed-use crawlers (Googlebot, Bingbot, etc.). Cloudflare says this option “will stop Applebot, Bingbot, and Googlebot from reaching your site — search included.” (Similarly, if you choose “block on pages with ads”, those pages won’t be included in web search.)

So to block AI training but remain in web search, choose disallow AI training.

As part of this change in mid-September, the setting options changed as well so check your current configuration. Earlier training settings of Block or Block on pages with ads became Disallow AI Training. Sites using the legacy Block AI bots setting were migrated to Search = Allow, Training = Disallow AI Training, and Agent = Block on pages with ads.

Another confusing thing about all of this is that since Microsoft’s Bingbot is a “mixed use” crawler (for both AI training and web search) and hasn’t yet launched an AI training proxy to block like Google has with Google-Extended (Cloudflare says Microsoft is targeting early 2027), the primary way to opt-out of Microsoft’s AI training is to add a noarchive meta tag to pages and when you configure these Cloudflare settings, Cloudflare doesn’t add that meta tag. So you’ll still need to set that up separately. (Cloudflare also mentions using Bing webmaster tools block URLs or content removal tools.)

They enforce these rules at the edge (at their server before passing the request to the site), not via robots.txt. The one setting that relies on robots.txt is Disallow AI Training for mixed-use crawlers: Cloudflare can’t block Googlebot for training only, so it lets Googlebot through and adds a no-training line to your robots.txt (for instance, a disallow for Google-Extended). This means that for this configuration to work, you (likely) need to have Bot Preference Sync enabled (see more on this below).

Bot Preference Sync – this enables Cloudflare to edit your robots.txt file to mirror your crawler settings. It’s turned on by default for new customers.

Accountable Label – Cloudflare has created this label for crawlers who commit to meeting these requirements:

• a way to opt-out of AI training (such as via robots.txt)
• a way to opt-out of AI summaries
• URL-level visibility into what is made available for training
• assurance that opting out of AI training doesn’t negatively impact web search rankings

They are using this labeling in how to handle crawler configuration. (For instance, they are allowing “mixed use” crawlers Googlebot, Bingbot, and Applebot when “disallow AI training” is set because they all have agreed to work towards this.)

402 payment required – Cloudflare provides a mechanism to return a 402 (payment required) and a price, but I couldn’t find any evidence that any crawlers were actually paying to crawl sites (and Cloudflare seems to be moving away from this to a “pay per use”, which I also didn’t find any evidence was being used).

For AI answers built through grounding and fan-out (see more on this below), Cloudflare’s settings mean that:

• if you choose Disallow AI Training (which adds a robots.txt disallow for Google-Extended), that also opts the site out of Gemini grounding.
• if the Agent setting is configured to block pages (or pages with ads), those pages may be returned in a response but the agent won’t be able to open or read the pages.
• if you set search to block, the pages won’t be in the web index that the grounding/fan-out queries are being run on, so won’t be in the potential consideration set.

Troubleshooting: Why Isn’t My Site Showing Up in LLM Search?

If your site isn’t showing up in AI answers the way you expect, a block may be somewhere you didn’t set it. Check these places, in this order:

1. The live robots.txt file on your domain, not the copy in your CMS or code repository. A CDN can add lines to it (Cloudflare’s Bot Preference Sync does).
2. Your CDN or WAF bot settings (in Cloudflare, Security > Settings > Configure AI bot policies).
3. Your host. When I checked this site in October, my host was returning a 403 to GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bytespider, including robots.txt, even though my robots.txt allowed everything. My host documents its list of blocked user agents and how to opt out, so yours may as well.
4. Meta robots tags and X-Robots-Tag headers: noindex, nosnippet, and noarchive.
5. The Search Console generative AI setting (Settings > Search generative AI), including whether the property inherits it from a parent property.
6. A real request using each crawler’s user agent. A robots.txt checker can’t see a block at the server or CDN; only a request can. (You can use my LLM crawler check tool for this, although note that if your sites are on Tiger Tech (the host I use), this checker won’t work because the checker can’t see Tiger Technologies’ block, because Tiger Tech doesn’t apply the block to requests from its own servers; this is yet another examples of how it can be complicated to find the source of visibility issues).
7. Your server logs, to see which crawlers actually reach you and what response they get.

To check robots.txt and the server response together, use my AI crawler access check, which requests the page with each crawler’s user agent. You can also use a web tool like crawler check to test robots.txt rules, a desktop tool like Screaming Frog to test server response codes, or ask your favorite LLM (like Claude Code) to run an analysis.

Proposed Standards: Content-Signal, Content-Usage, and DisallowAITraining

New standards are still evolving and they take time. In part because LLMs and therefore what standards are needed are still evolving, and in part because it’s hard to get organizations to agree (not everyone thinks the same standards are needed; every company may have a different approach and perspective). (As someone involved with the Google, Microsoft, and Yahoo XML Sitemaps alliance, I can attest that it’s not easy!)

Here are a few proposed robots.txt exclusion protocol standards that have not (yet at least) been adopted (as with other robots.txt directives, each of these could be set for the entire site or specific directories):

• content-signal: Cloudflare has proposed this extension (and is putting it in robots.txt files) with three options that can have a yes or no flag: ai-train (use in LLM training models), ai-input (user-triggered LLM access), and search (use in web search indexing). The directive that would exclude LLMs from using the site in training data is content-signal: ai-train=no.
  
  As far as I can tell, no LLMs have said they support it. Google’s John Mueller has said that “none of the crawlers/LLMs” use it and “it has no effects whatsoever for any crawler or LLM”.
  
  It is frustrating that Cloudflare’s blog post announces they are adding this functionality as though it’s something that will actually benefit site owners today and is not straightforward about the fact that at least right now, it won’t actually do anything. On the one hand, the post says “Our customers are fully in the driver’s seat for what crawlers they want to allow and what they’d like to block.” But on the other hand, it also says “Signaling the rules of the road only works if others recognize those rules.” Which in addition to not being clear that currently this directive does nothing, makes it seem like any company not following the directive is not “following the rules of the road”, when the reality is that the rules of the road haven’t yet been established and Cloudflare is making up the rules on their own.
  
  But anyway, their proposal is that these lines in a robots.txt file would allow all bots to crawl the entire site for web search indexing but not LLM training:

User-Agent: *
Content-Signal: search=yes ai-train=no
Allow: /

• DisallowAITraining: Microsoft had proposed this similar extension to allow for web indexing but not AI training. IETF shows the proposal as expired (and the name changed). It was authored in part by Fabrice Canel, who has been in charge of Microsoft’s web search crawling (and more over the years) since long before Microsoft’s search engine was called “Bing”. The latest from Microsoft is that they expect to have robots.txt support for blocking crawling for AI training in early 2027.
• content-usage: Google and Mozilla have proposed this extension to allow crawling for web indexing but not AI training. This differs from their support of blocking the user-agent Google-Extended in that content-usage (similar to content-signal) is a proposed standard that all LLMs would follow (which would negate the need to block each user-agent separately). The same Mozilla author is part of another working group document as well and both are part of the standards proposed by the IETF AI Preferences (AIPREF) working group.

Additional Questions

What is RAG (retrieval-augmented generation)?

Retrieval-augmented generation is the process LLMs use to augment their training data with information from real-time (or at least at web search indexing-time) web pages. They combine input from both training data and web search results in their responses. They use training data to inform what web searches to do. For instance, if the user is asking about running shoes, the agent may generate a list of trusted brands from the training data and based on that, do web searches about those brands.

People refer to these web searches as “grounding queries” and “fan-out queries,” since they’re searches the LLM chooses to run based on the user’s prompt rather than the user’s own search. “Grounding” is Google’s term for retrieval-augmented generation, and “query fan-out” is Google’s term for the related searches the model generates to get more results for a response. Gemini runs its fan-out queries against the Google web index. There’s no site crawling when this happens; the search engine or LLM searches an index it has already built. You can watch an LLM such as ChatGPT or Claude do these searches during a session.

Do LLM crawlers read .md versions of pages or llms.txt files?

Not the way many people expect. In Dries Buytaert’s logs, no AI crawler asked for markdown in place of HTML, but some fetched .md files that the HTML pages linked to, along with the HTML itself. The requests for markdown come from coding agents such as Claude Code. There’s no measured benefit yet: in a controlled test by Profound, markdown pages got about one extra bot visit, which wasn’t statistically significant, and an Ahrefs study found that 97% of llms.txt files never get read. Google says Search doesn’t use llms.txt files or markdown versions.

What changed

Oct 2026

First published.