E1 · Run and open
15 minRun the auditor on your site (or a client's, with notice) with --max 50 and --saida.
Done when: the folder has relatorio.html, auditoria.json, achados.csv and paginas.csv, and the report opens in the browser.
Lesson 2 · 18 min
auditor: the X-ray of the whole site
The X-ray from course 1 looks at one page at a time. The auditor crawls the site politely, like a bot that doesn't run JavaScript, and checks every page: canonical, hreflang, sitemap, orphans, robots.txt by AI bot purpose, and CDN blocking. Each finding comes with a priority and a one-line fix.
Start the lesson
In this lesson you will
- 1How the auditor crawls without weighing on the server
- 2What it checks on each page and across the site
- 3Reading robots.txt by purpose and the CDN test
- 4The honest limit: partial crawl and what's left out
Goal: run the auditor on your site and sort the findings by priority, with the fix line for each one.
01The auditor's path
GET only, one request per second, at most the pages you ask for. Click each step.
Click each block in the diagram to see what it does.
It crawls slowly, reads like a bot, and gives you a list in priority order.
02Run, read, prioritize
Click each step.
1. Run
Run it on the home page with --max 50 and an output folder. With a 1 s pause, a small site takes a minute or two.
Use -q to avoid listing every page in the terminal.
Etiquette
The minimum pause is 0.5 s and each response is cut off at 2 MB. On a client's site, give notice before you run it.
2. Critical
Critical is what takes pages out of the search engine: robots.txt with a 5xx error (it becomes everything blocked), a blanket Disallow, a blocked search bot, a broken internal link, noindex on a sitemap URL.
If there's anything critical, stop and fix it before looking at the rest.
3. Important
Canonical missing or multiple, hreflang without a return link, sitemap without lastmod, a page with fewer than 80 words in the HTML.
Improvement (long title, no JSON-LD, image without alt) goes into the month's queue.
Not a Google rule
The title range of 15 to 65 characters comes from webmcp-readiness. Google cuts off by width; treat it as a reminder, not an error.
4. Training bot
GPTBot, ClaudeBot, and Google-Extended blocked show up as info: blocking training doesn't take the site out of search.
A search bot blocked, in robots or at the CDN, is critical: you disappear from that assistant's answers.
03The auditor commands
Run from the ferramentas/auditor folder of inemaseo-sistemas.
Terminal · your site
cd ferramentas/auditor python3 auditor.py https://yoursite.com --max 50 -q --saida auditoria/yoursite
Terminal · just a guide's folder on GitHub Pages
python3 auditor.py https://username.github.io/repo/guia/ --max 20 --saida auditoria/guia
Terminal · critical and important findings from the CSV
grep -E '^(critico|importante),' auditoria/yoursite/achados.csv
Example from the repository
python3 auditor.py https://inema.club --max 50 (without -q, shortened list):
[ 1/50] 200 https://www.inema.club/ [ 2/50] 200 https://www.inema.club/en/ (…) [50/50] 200 https://www.inema.club/conhecimento/metodologia-de-ensino-do-inema/ www.inema.club: 50 páginas, 0 crítico(s), 1 importante(s), 2 melhoria(s). Relatório: …/relatorio.html [importante] Pouco texto no HTML entregue (< 80 palavras)
In achados.csv, the important finding points to two pages: /conhecimento/curso-atia/ (77) and /conhecimento/curso-codex-basico/ (78). The JSON records 62 requests and 1,287 URLs still in the queue.
From the auditoria.json of the same crawl:
busca Googlebot, Bingbot, OAI-SearchBot,
Claude-SearchBot, PerplexityBot → liberados
usuário ChatGPT-User, Claude-User,
Perplexity-User → liberados (grupo *)
treino GPTBot, ClaudeBot, Google-Extended → liberados
CDN OAI-SearchBot 200 · PerplexityBot 200
Perplexity-User 200 · GPTBot 200On a GitHub Pages guide (rea-engenharia-reversa/guia/), the repository example shows "robots.txt não existe": the robots.txt that counts is the one at the host root, which belongs to another repository.
- Reading "orphan" as absolute truth: with --max 50 on a site with 1,000 URLs, orphan means "no link from the 50 pages read."
- Thinking it measures speed: Core Web Vitals and backlinks are left out; use PageSpeed and URL Inspection.
- Trusting only the CDN test: it sees rules by User-Agent; a CDN that checks the real bot's IP may act differently.
04Practice: audit your site
Keep the output folder: in lesson 7 you compare month by month.
E2 · Three findings in order
20 minPick the 3 highest-priority findings (if there are no critical or important ones, the first 3 improvements) and note the URL and the line on how to fix it.
Done when: a list of 3 lines in priority order, each with the URL, the finding and the fix copied from the report.
05Test what you learned
The explanation shows up after your answer.
Question 1 of 4
Your site's robots.txt returns a 503 error. How does the auditor treat it?skill
Correct
Per RFC 9309, a 5xx error on robots.txt is treated as everything blocked; a 4xx (file doesn't exist) is treated as no restriction. That's why 5xx is critical.Not that one
Per RFC 9309, a 5xx error on robots.txt is treated as everything blocked; a 4xx (file doesn't exist) is treated as no restriction. That's why 5xx is critical.Review: Run, read, prioritizeQuestion 2 of 4
The report says GPTBot is blocked. What's the priority?
Correct
GPTBot is for training. The one that searches and cites in ChatGPT is OAI-SearchBot. Blocking training is the owner's choice; blocking search is critical.Not that one
GPTBot is for training. The one that searches and cites in ChatGPT is OAI-SearchBot. Blocking training is the owner's choice; blocking search is critical.Review: Run, read, prioritizeQuestion 3 of 4
With --max 50 on a site with 1,000 URLs, a page shows up as an orphan. What does that mean?
Correct
The crawl is partial. The report says how many URLs stayed in the queue; raise --max or check the links before concluding.Not that one
The crawl is partial. The report says how many URLs stayed in the queue; raise --max or check the links before concluding.Review: The auditor commandsQuestion 4 of 4
Why doesn't the auditor run JavaScript?
Correct
Google renders JavaScript; to see what it sees, use URL Inspection. Many other bots may not render, and the auditor shows that view.Not that one
Google renders JavaScript; to see what it sees, use URL Inspection. Many other bots may not render, and the auditor shows that view.Review: The auditor's pathQuiz complete
You got {a} of {n} right. Get them all right and the lesson is marked as complete.Pick an option. The explanation appears after you try.
06Takeaways
PoliteGET only, 1 s pause, --max pages, respects robots.txt.
OrderCritical, important, improvement. A blocked training bot is info.
LimitPartial crawl; CDN checked by User-Agent only; no speed or backlinks.