E1 · The example by purpose
5 minRun the example robots.txt block and open kit/site/robots.txt to see the comments for each group.
Done when: python3 robots.py site/robots.txt → treino line with GPTBot bloqueado and saída: 0.
Lesson 6 · 15 min
Who's arriving: bots by purpose and signed agents (the concept)
Not every AI bot that visits your site does the same thing: some index for search, others read when a person asks, others collect for training. The robots.txt file asks by purpose, but it doesn't prove who is who. In this lesson you read your robots.txt by purpose and learn what the Web Bot Auth draft proposes for identifying the agent.
Start the lesson
In this lesson you will
- 1Search, user action and training: three purposes
- 2The robots.txt as read by the kit's robots.py
- 3Why the name in the User-Agent proves nothing
- 4Web Bot Auth and signed agents, as a concept
Goal: read your robots.txt by purpose and understand why only a signature on the request identifies the agent.
01Who knocks at the door, and how to tell
Click each block.
Click each block in the diagram to see what it does.
Purpose is decided in robots.txt; identity only with a signature. Security in depth for this is course 4.
02Request × identity
What each one solves.
1. robots.txt
One group per User-agent with Allow and Disallow. Bots that follow the rule respect it; the CDN or firewall is what enforces it.
Write the decision by purpose and note the reason in a comment. The full step-by-step is in lesson 6 of the inemaSEO Foundation course.
2. User-Agent
Each bot introduces itself by name in the User-Agent header. It's just text: any program can write "GPTBot". It's good for statistics, not for trust.
3. Signature
In the draft-ietf-webbotauth-httpsig-protocol draft (Sept/2026, Cloudflare and Google, an IETF working group document, not yet an RFC), the agent signs the request and sends the Signature-Agent header; the keys live in a JWKS directory at a well-known URI.
On Cloudflare, since 07/01/2026, signed agents count as Verified, with the field "Direct versus Intermediary access".
What changes for you now
Nothing to install in this course. Knowing it exists helps you read your CDN dashboard and not trust a bot's name alone.
03The robots.txt read by purpose
The robots.py reads a robots.txt (file or URL) and tells you, by purpose, who can get into the root of the site. Simplified rule: only the root "/"; partial = blocks some paths.
Terminal · the example robots.txt
cd ~/projetos/formacao-webmcp-base-v72/kit python3 robots.py site/robots.txt; echo "saída: $?"
Template · your site
cd ~/projetos/formacao-webmcp-base-v72/kit python3 robots.py https://<your-site>/robots.txt; echo "saída: $?"
Example from the repository
grupos no arquivo: *, chatgpt-user, claude-searchbot, claude-user, claudebot, google-extended, gptbot, oai-searchbot, perplexity-user, perplexitybot busca OAI-SearchBot liberado · Claude-SearchBot liberado · PerplexityBot liberado · Googlebot parcial (*) · Bingbot parcial (*) ação do usuário ChatGPT-User liberado · Claude-User liberado · Perplexity-User liberado treino GPTBot bloqueado · ClaudeBot bloqueado · Google-Extended bloqueado lembrete: robots.txt pede, não identifica. Quem prova quem é: assinatura na requisição (Web Bot Auth, rascunho IETF). saída: 0
The example shop allows search and user action and blocks training. Googlebot and Bingbot don't have their own group: the rules of * apply, which blocks only /admin/ (that's why "parcial").
grupos no arquivo: *, bingbot, claude-searchbot, claudebot, googlebot, oai-searchbot, perplexitybot busca OAI-SearchBot liberado · Claude-SearchBot liberado · PerplexityBot liberado · Googlebot liberado · Bingbot liberado ação do usuário ChatGPT-User liberado (*) · Claude-User liberado (*) · Perplexity-User liberado (*) treino GPTBot liberado (*) · ClaudeBot liberado · Google-Extended liberado (*) lembrete: robots.txt pede, não identifica. Quem prova quem é: assinatura na requisição (Web Bot Auth, rascunho IETF). saída: 0
inema.club allows everything, including training: it's open content and chose to allow it. The user-action bots don't have their own group and fall under *.
04Practice
The example, your robots.txt, and the decision noted.
E2 · Your robots.txt
10 minRun the template with your site's address. Copy the busca, ação do usuário, and treino lines into the plan (the "Quem pode entrar" section).
Done when: the output has the busca, ação do usuário, and treino lines and ends with saída: 0.
E3 · What really counts
10 minOpen your CDN or hosting dashboard and check whether there is AI bot blocking or a list of verified agents. Note whether it agrees with the robots.txt.
Done when: the plan has a line saying which CDN (or "no CDN") and whether the dashboard agrees with the robots.txt.
05Lesson quiz
Question 1 of 4
A person pastes your site's link into the assistant and asks for a summary. What kind of bot reads the page?skill
Correct
User-action bots visit the page at someone's request. Blocking them stops the assistant from reading your site even when the customer asks.Not this one
User-action bots visit the page at someone's request. Blocking them stops the assistant from reading your site even when the customer asks.Review: Who knocks at the door, and how to tellQuestion 2 of 4
Your log shows User-Agent "GPTBot". Does that prove it was OpenAI?skill
Correct
The name declares, it doesn't prove. Proof comes from a signature on the request (Web Bot Auth, draft) or from the verification the CDN does.Not this one
The name declares, it doesn't prove. Proof comes from a signature on the request (Web Bot Auth, draft) or from the verification the CDN does.Review: Request × identityQuestion 3 of 4
What does the Web Bot Auth draft propose?
Correct
HTTP signature (RFC 9421), the Signature-Agent header, and keys in a JWKS directory at a well-known URI. An IETF draft, not an RFC.Not this one
HTTP signature (RFC 9421), the Signature-Agent header, and keys in a JWKS directory at a well-known URI. An IETF draft, not an RFC.Review: Who knocks at the door, and how to tellQuestion 4 of 4
The robots.txt allows everything, but an agent can't read the site. Where do you look first?
Correct
What really blocks is the CDN or the firewall. If the dashboard blocks what robots.txt allows, the dashboard wins.Not this one
What really blocks is the CDN or the firewall. If the dashboard blocks what robots.txt allows, the dashboard wins.Review: Request × identityQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as done.Pick an answer. The explanation shows up after you try.
06In three sentences
Three purposesSearch, user action, and training: decide one by one.
A request is not proofrobots.txt asks, User-Agent declares, the CDN enforces.
Signing is the wayWeb Bot Auth signs the request; it's still a draft.