E1 · The sample
10 minRun the analysis on the mock sample and find in the report the fake bots and the page that no bot read.
Done when: you noted which bots had fake hits and which sitemap page went unread.
Lesson 6 · 17 min
botlog: who really reads your site
The robots.txt tells you what you allow; the server log tells you who showed up. botlog reads the access log, sorts the AI bots by purpose (search, user request, training), checks each one's IP against the company's official list, and shows you the fake ones. And it filters the log before you store or show it, because a person's IP is personal data.
Start the lesson
In this lesson you will
- 1The four purposes and what happens if you block each one
- 2Fake bot: why the User-Agent isn't enough
- 3Run the analysis on an nginx or Cloudflare log
- 4Filter the log before storing it (LGPD)
Goal: sort the AI bots in the log by purpose, spot the fake ones, and filter the log before storing or sharing it.
01From the log to the list of who read
Each log line goes through four questions. Tap each one.
Click each block in the diagram to see what it does.
The log shows who requested the page, not whether the AI used the content. For citation, lesson 5.
02Read the report
Tap each point.
1. Purpose
Blocking AI search = disappearing from that assistant's answers with sources. Blocking the user's agent = the AI doesn't open the page when someone asks.
Blocking training takes the site out of the training of future models, without disappearing from search.
2. Fakes
Anyone can write GPTBot in the User-Agent. Without checking the IP, the count inflates.
The report shows the IP only for the fake bots, so you can block them if you want.
3. 403 for AI
An authentic AI bot getting a 403 is usually a CDN or WAF block, not robots.txt.
Since 09/15/2026, free Cloudflare blocks training bots and agents by default on pages with ads: check your site's dashboard.
4. LGPD
The IP of someone who is not a bot is personal data. Before you show, publish, or keep a log for a long time, run filtrar: only the bot lines with an official IP stay.
An old list makes a false "fake": the Bing one in the folder is from Jan 2024. Update the listas/ folder.
03Analyze and filter
Run it from the ferramentas/botlog folder. The MONTADO sample uses official User-Agents and IPs from documentation ranges (RFC 5737), which belong to nobody.
Terminal · analisar (several files, .gz included)
cd ferramentas/botlog python3 botlog.py analisar ../../exemplos/botlog/entrada/MONTADO_access.log ../../exemplos/botlog/entrada/MONTADO_cloudflare.jsonl \ --sitemap ../../exemplos/botlog/entrada/MONTADO_sitemap.xml --saida saida/montado
Terminal · your server (read-only) and your sitemap
ssh root@YOUR_SERVER 'cd /var/log/nginx; zcat -f access.log.*.gz; cat access.log.1 access.log' > /tmp/site.log python3 botlog.py analisar /tmp/site.log --sitemap https://yoursite.com/sitemap.xml --saida saida/site
Terminal · LGPD: only bots with an official IP
python3 botlog.py filtrar /tmp/site.log --saida /tmp/bots-only.log rm /tmp/site.log # the full log doesn't need to stay
Example from the repository
26 lines, 22 from bots (training 6, search 5, search engine 4, user 3); fake 4; never visited 1/6 → saida/montado/relatorio.html
From robos.csv (mock sample):
bot purpose hits genuine fake errors GPTBot training 5 3 2 2 ClaudeBot training 3 3 0 2 Googlebot search engine 3 2 1 0 OAI-SearchBot search 3 3 0 0 PerplexityBot search 2 1 1 0
ClaudeBot's 2 errors simulate a CDN block (403).
nginx log from desenho.inema.pro, Oct 5 to Oct 19, 2026, already run through filtrar (the original had 3,082 lines; 8 were left, only bots with an official IP):
8 lines, 8 from bots (training 6, search 2); fake 0; never visited 0/1 → saida/real/relatorio.html
And filtrar on the mock sample:
16 of 23 line(s) kept (only bots with an official IP) → /tmp/bots-only.log
- Counting by User-Agent: without checking the IP, a fake bot becomes an "AI hit".
- Looking for Google-Extended in the log: it doesn't show up; it's a robots.txt token.
- Looking only at the server log behind a CDN: it sees only what got through the CDN, sometimes with the CDN's IP. Use the CDN log.
- Publishing the whole log: a person's IP is personal data; filter first.
04Practice: who reads you
No access to the server log? Do E1 and ask your CDN or host for the log for E2.
E2 · Your log, filtered
30 minCopy your server's log (read-only), run analisar and then filtrar; delete the full log.
Done when: you have the report by purpose, the "N of M line(s) kept" line from filtrar, and the full log deleted.
05Test what you learned
The explanation shows up after your answer.
Question 1 of 4
The log shows 50 hits from "GPTBot". What else do you need before you believe it?skill
Correct
Anyone can write GPTBot in the User-Agent. botlog checks the IP against the published list; outside it, it's fake and doesn't count.Not that one
Anyone can write GPTBot in the User-Agent. botlog checks the IP against the published list; outside it, it's fake and doesn't count.Review: From the log to the list of who readQuestion 2 of 4
You block GPTBot and ClaudeBot. What happens?
Correct
They are training bots. The ones that search and cite are OAI-SearchBot and Claude-SearchBot; the ones that open a page on request are ChatGPT-User and Claude-User.Not that one
They are training bots. The ones that search and cite are OAI-SearchBot and Claude-SearchBot; the ones that open a page on request are ChatGPT-User and Claude-User.Review: Read the reportQuestion 3 of 4
A genuine ClaudeBot gets a 403 in the log. Most likely cause?
Correct
robots.txt is a request; a polite bot doesn't even ask for the page. A 403 is a response from the server or the CDN, which enforces it. Check the CDN dashboard.Not that one
robots.txt is a request; a polite bot doesn't even ask for the page. A 403 is a response from the server or the CDN, which enforces it. Check the CDN dashboard.Review: Read the reportQuestion 4 of 4
Why run filtrar before keeping the log?
Correct
A verified bot is not a person. The rest of the log has visitor IPs, which should not be published or stored without need.Not that one
A verified bot is not a person. The rest of the log has visitor IPs, which should not be published or stored without need.Review: Analyze and filterQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after you try.
06Takeaways
PurposeSearch, user, training, and crawler: blocking each one costs different things.
official IPUser-Agent proves nothing; a fake bot doesn't count.
LGPDfilter before storing or showing; delete the full log.