Lesson 6 · 18 min
"Block the AIs" has become a common request, and it is almost always worded wrong. Each AI company has different robots for searching, for opening a page at someone's request, and for training models. Blocking one does not block the others. And in 2026 your CDN may be deciding for you.
Start the lesson
Goal: decide in robots.txt and in the CDN dashboard what to allow for search, for user action, and for training.
Tap each block.
Decide by purpose, write it in robots.txt, and check that your CDN agrees.
Click on each one.
For almost every business, yes: allow the search robots. They are how your site gets into the answers.
Blocking search to "protect your content" is usually the worst of both worlds: the text has already been read, and you disappear from the answers.
If a customer pastes your link into the assistant and asks for a summary, this is the bot that reads it. Allowing it is usually the natural choice if you sell or teach.
There's no right answer here. It's a business choice. Blocking training does not take you out of search.
Write down the reason in the robots.txt itself, as a comment, for whoever comes next.
The robots.txt also allows ClaudeBot, which is a training bot: INEMA is open content and chose to allow it.
Open the CDN dashboard and look for the bot and AI settings. Check whether a default block is turned on, and on which pages.
If the dashboard blocks what the robots.txt allows, the dashboard wins.
Adapt the training decision to yours. Change the sitemap address.
# robots.txt by purpose (adapted from the inemaSEO templates/robots.txt) User-agent: * Allow: / # Search: show up and get cited in answers User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Bingbot Allow: / # User action: the AI opens the page at someone's request User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Allow: / # Training: your call. This example blocks; blocking training does not take you out of search User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: / Sitemap: https://www.seusite.com/sitemap.xml
# Your Site > One sentence saying what it is, who it is for, and what it delivers. ## Main pages - [Home](https://www.seusite.com/) - [About / who runs it](https://www.seusite.com/sobre/) - [Catalog](https://www.seusite.com/catalogo/) ## Data for machines - [Sitemap](https://www.seusite.com/sitemap.xml)
Bot decision for <seusite.com> on <mm/dd/yyyy> - Search: allowed (reason: <...>) - User action: <allowed/blocked> (reason: <...>) - Training: <allowed/blocked> (reason: <...>) - CDN: <name>, default AI block <on/off>, checked in the dashboard - llms.txt: <yes/no> (optional)
The method's templates/robots.txt puts ClaudeBot together with the search bots. According to Anthropic's documentation, ClaudeBot collects data for training; search belongs to Claude-SearchBot, and reading on request belongs to Claude-User. The model above separates the three and adds the user action bots.
Ahrefs study (15/06/2026), 137 thousand domains: among valid llms.txt files, 97% received no requests at all. Google's AI guide says you can ignore it.
Honest take: it's cheap and can work as an index for coding agents. It is not a citation lever, and it should not be sold as "get into ChatGPT".
Made-up conversation for practice. Pick the answer you would give; the comment appears right after.
Choose your answer. The comment appears right after, and the conversation continues based on your choice.
Use the robots.txt block from your raio-x.txt.
Rewrite your robots.txt with the search, user action, and training groups, with a comment saying the reason for the training decision. Publish it and run the X-ray again.
Done when: the robots.txt block in the X-ray shows the three groups and the Sitemap at the end.
Open your CDN or hosting dashboard and check whether AI bot blocking is turned on. Fill in the decision note.
Done when: the note says which CDN, whether the default block is on, and whether it agrees with robots.txt (or "no CDN").
The explanation appears after you answer.
Question 1 of 4
You blocked GPTBot, ClaudeBot, and Google-Extended. What happens to search?skill
Question 2 of 4
robots.txt allows everything, but an agent can't read the site. Where do you look first?
Question 3 of 4
What should you say about llms.txt?
Question 4 of 4
The client wants to "block all AIs." What is the first question?
Pick an answer. The explanation appears after you try.