May AI systems read your site

robots.txt in one minute

It is a text file at uwdomein.be/robots.txt. Publicly readable, so you can open your competitor's just as easily as your own.

It is a request and not a lock. Well-behaved crawlers keep to it. A scraper that ignores it is not stopped by it. If you really want to shield something, you need a password or a firewall rule. Beyond that robots.txt governs whether a bot may read and not whether your page enters the index. That distinction is in chapter 1.

One technical point breaks most files. Rules apply per User-agent block and a bot reads only the block that fits its name best. So it does not read both its own block and the one for User-agent: *. If you tidily put a Disallow: /admin/ under the asterisk and then add a block for GPTBot with only Allow: /, GPTBot may enter your admin screen. Repeat your Disallow rules in every block where they have to apply. In the example further on that is done by putting several User-agent lines above one set of rules.

Three kinds of access that almost everyone confuses

Training. The material a model is taught with. If you block this, your text does not enter a next version of the model. It does not mean that you disappear from the answers, because answers about current subjects nowadays come largely from a search at that very moment.

Display in the search results of the assistant. This is the bot that fetches your page while the assistant is searching. If you block this one, the assistant cannot cite you. This is the access that counts if you want to be mentioned.

Fetching after a click. The user pastes your link or asks the assistant to summarise your page. If you block this one, your own visitor gets the message that the page cannot be read.

Block only the first group and you stay quotable. Block everything and you are gone from the answers.

The crawlers by name

The names Claude-Web and anthropic-ai were the old names of Anthropic. Since February 2026 they have been withdrawn and replaced by the three above. They may come out of your file and if you leave them in they do no harm because nothing comes past under that name any more.

Two notes with this list. It changes: suppliers add names, rename bots and do not always describe the role equally sharply. Put it in your diary twice a year to read this file over. And do not base a strict choice on a role you read here without testing it against the documentation of the supplier itself.

Googlebot is deliberately absent. It falls under User-agent: * and needs no block of its own. Google-Extended is a separate name that governs only the AI use and changes nothing about your ordinary position in Google.

A robots.txt to take over

Replace the example paths and the sitemap URL. The rest can stay as it is.

# robots.txt for uwdomein.be
# Search engines and AI assistants may read and cite this site.
# Last checked: 3 August 2026

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /klantenzone/

# Classic search engines
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: YandexBot
Allow: /
Disallow: /admin/
Disallow: /klantenzone/

# AI assistants: display in the results and fetching after a click
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Applebot
Allow: /
Disallow: /admin/
Disallow: /klantenzone/

# Training data
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /admin/
Disallow: /klantenzone/

Sitemap: https://uwdomein.be/sitemap.xml

If you do want to be cited and not be trained on, replace Allow: / in the last block with Disallow: /. The rest stays as it is. That is the whole intervention.

Our own choice is to allow everything. We publish the guide to be read and a model that may not read our text never cites us. For a publisher who lives off licences that trade-off lies differently. There is no generally right answer here.

Google AI Overviews and why robots.txt is not about that

If you block Google-Extended, you simply stay in the search results of Google. Googlebot is a different bot and you do not lose it with this.

What Google-Extended exactly does and does not touch within the AI parts of Google Search is not fully transparent. If you want to stay out of an AI Overview with certainty, the only hard switch is the snippet control: nosnippet in your robots meta or data-nosnippet around a piece of text. That also switches off your ordinary snippet in the search results. That is an expensive price.

And it is a price for a problem you may not have. In our own baseline measurement of 2 August 2026 not one of fifteen Dutch-language searches in Belgium produced a Google AI Overview. Zero out of fifteen. In that language and that region the layer therefore barely exists today. For a Flemish or Dutch SMB the discussion about AI Overviews is thereby theoretical for now. In English the picture is different.

Our robots meta therefore stays index, follow, max-snippet:-1, max-image-preview:large. That is the reverse of nosnippet: you give Google and the assistants all the room to show you. Anyone who wants to be cited does not squeeze that.

The silent block: security plugin, CDN and firewall

In our experience this is the most common cause of invisibility in AI answers. It is almost never a deliberate choice. It is a setting that someone once switched on or that was on by default.

Where it comes from:

• A security plugin with a list of barred bots. That list is kept up by the maker and not by you.
• A CDN or firewall with bot management. Some providers have a switch with a name like block AI scrapers and it is sometimes on by default.
• A check page that first wants to run JavaScript before the real content arrives. A crawler that does not run JavaScript gets that page and nothing else.
• Rate limiting that throttles a crawler after a few requests.
• A country block. Crawlers do not necessarily come from your country.
• A cookie wall that returns a consent page instead of your text.

The treacherous part: your robots.txt says Allow: / and is entirely right. The block sits one layer higher, before your site even begins. A robots.txt checker reports nothing, because it only reads the file.

What we found on our own site

This chapter is not theoretical. On 4 August 2026 we looked at the crawler log for semanu.be. In twenty-four hours: 229 requests from AI crawlers, 107 let through and 122 failed. Of those 122, one bot accounted for 120.

Our robots.txt allowed eighteen crawlers explicitly at that moment, PerplexityBot among them. The file was correct. The block sat at our CDN: a switch called Bot fight mode was on. It challenges anything not recognised as a verified bot, and Cloudflare had removed Perplexity from that list in August 2025 after a dispute over how their crawler behaved.

The result: one of the four large answer engines could not read our site while our robots.txt was welcoming it. That immediately explained why that engine never mentioned us in our own visibility measurement. Not a content problem and not a writing problem, but a checkbox.

The lesson you can take from this: robots.txt is a statement of intent and the layer in front of it decides. Only the crawler log of your CDN or your server shows what actually happens.

How you check whether it works

The third check is the only one that catches the silent block. One line in a terminal:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "OAI-SearchBot" https://uwdomein.be/

You want to see 200. If you get 403, 429 or 503, something is holding it back. Repeat for GPTBot, ClaudeBot and PerplexityBot.

A 200 is not automatically good news, because a check page gives one as well. So also look at what comes back:

curl -s -A "ClaudeBot" https://uwdomein.be/ | head -c 400

If you see your own headings and text, it is in order. If you see a security message or an empty page with only a script, something sits between you and the bot.

What robots.txt does not govern

It does not take you out of answers that already exist. What a model has learned it does not forget because you add a line today.

It does not stop a scraper that pays no attention to it.

It does not decide whether you get indexed. That is chapter 1.

And it does not produce citations. Access is the condition and not the outcome. Chapter 6 sets out the four factors and access is one of them. Whether it works you measure as described in chapter 9.

What you can do yourself and what you cannot

To do yourself, without a developer:

• Open and read uwdomein.be/robots.txt. Five minutes and it is the cheapest check in this whole guide.
• Take over the file above and adjust the Disallow paths and the sitemap URL.
• Run the curl test for four bot names. On Mac and Linux curl is already there and on Windows it sits in PowerShell.
• Look at the robots.txt report of Search Console.
• Search the control panel of your CDN or security plugin for a switch about bots.

What you need help with:

• A bot management rule at your CDN that only your host can switch off.
• A firewall that blocks on the basis of behaviour instead of on name. No rule in robots.txt helps against that.
• A site that only builds the content through JavaScript. Then access is not the problem and readability is.
• Access to the server log if you do not have it. Ask your host for an extract with the bot names in it.

The whole exercise is half an hour. Together with chapter 1 it is the only part of this guide where one setting can make the difference between existing and not existing for an AI answer engine.

Would you rather not do this yourself: SEMANU carries out the full SEO and GEO audit for 1,450 euro, report and working session included. What it covers is on the guide page.