Detailed content
🎯 The goal
The target is a a watchdog that never sleeps: a worker that reads forums, communities, and feeds, identifies recurring complaints and wishes—the “pain points”—and gives you a list of what could become a product, ranked by potential and linked to the source. Instead of scrolling Reddit for hours, the local model does the heavy reading and only alerts you when it finds something real.
🆕 New here?
A "pain" is a problem people repeatedly complain about (“I waste time doing X by hand,” “no tool solves Y”). When the same pain point comes up often and no one solves it well, that’s a sign of opportunity — the raw material for a product. The watcher exists to capture these signals at scale.
🔑 The central idea
The watcher has four responsibilities, in this order:
- •Collect from permitted sources (RSS, official JSON, APIs) — without violating anyone’s rules.
- •Score for each item, ask the LLM: Is this a real pain point? What product would solve it? Rate it from 0 to 10.
- •Rank and deduplicate, keeping the source of each signal so you can verify it.
- •Run on its own every N minutes and only notify you about what passes the threshold.
⚖️ Ethics and rules (read before collecting)
Before writing any code: watching the web doesn't mean scraping everything at any cost. Every site has terms of service (ToS) and a file robots.txt that determine what you can access. Respecting them isn't just about "being nice"—it's what keeps your worker sustainable, avoids blocks, and keeps you within the law.
🚨 Alert box: the non-negotiable rules
- ✗Use the official API/feed when available. Reddit, YouTube, Stack Overflow, Hacker News, and many forums provide official JSON/RSS—always prefer that over scraping HTML.
- ✗Respect the ToS and robots.txt. If the terms prohibit automated collection, don’t collect. Platforms like X/Twitter impose strict restrictions—don’t try to force it.
- ✗Apply rate limits. Space out requests (e.g., 1 every 2–5 s); never hammer the server. Identify yourself with a User-Agent real and contact.
- ✗No personal data or third-party logins. Collect only public content and aggregate trends—don’t track individuals.
✓ Responsible data collection
- ✓Official RSS/JSON or API with a key
- ✓Identifiable User-Agent + rate limit
- ✓Local cache so you don’t fetch the same thing again
- ✓Public content only, keeping the source
✗ Problematic data collection
- ✗Bypass login, paywall, or CAPTCHA
- ✗Ignore robots.txt / ToS
- ✗Fire off thousands of requests/min
- ✗Collect personal data about individuals
💡 Practical tip
Start with what is explicitly offered: Reddit serves JSON on any page (just add .json) and RSS (.rss); Hacker News has the Firebase API; many blogs and forums have /feed. With official sources, you never have to wonder whether you're doing something wrong.
📥 Collecting from the permitted source
First concrete step: pull items from an official source and normalize everything into a single format — that way, the rest of the pipeline doesn't need to know where it came from. We'll use Reddit's official JSON as an example, writing one line per item (format JSONL).
⌨️ Copy-run: collect and normalize
Objective: download new posts from a subreddit through the official source and save one clean item per line. Needs curl e jq.
How to check: run wc -l coletado.jsonl (should show ~25) and head -1 coletado.jsonl | jq — every line has title, text e url of origin.
💡 Practical tip
Normalization is the key to combining sources without rewriting anything. Tomorrow, you can switch collection to an RSS feed (/feed) or through the Hacker News API, keeping the same keys (source, id, title, text, url). The LLM and ranking won't even notice the difference.
🧠 Analysis with the local model
Now for the heart of it: for each item collected, ask the local model to classify and return structured JSON. The key is to force JSON output (Ollama accepts format:"json") and use a low temperature for a consistent response. This turns loose text into data the ranking system understands.
⌨️ Copy-run: score an item via the local API
Objective: send an item to the local API (OpenAI-compatible, from Track 3) and receive {real_pain, pain, product, score}.
How to check: the output must be valid JSON with score between 0 and 10. If it returns unstructured text, reinforce the prompt with "respond ONLY with JSON" and confirm that it passed format:"json".
💡 Practical tip
Also request a field justification short: besides improving the score (the model "thinks" first), it gives you a line to read in the report without reopening the source. And give examples in the system prompt of what a score of 2 vs. a score of 9 means — it calibrates the score much more than adjusting the temperature.
📊 Ranking and saving
With each item scored, the worker becomes a curator: sorts by score, removes duplicates, cuts anything below the threshold, and saves only what’s worth keeping, always with the source link. Discipline here is what separates a useful report from a flood of noise.
Sort by score
Greatest potential at the top. sort_by(-.score) solve it in a single line of jq.
Deduplicate
The same pain point comes up in several posts. unique_by(.id) and ideally, group items with similar pain points.
Cut below the threshold
Keep only real_pain == true e score >= 7. The cutoff is your signal-to-noise knob.
Record with citation
Each opportunity carries the url of origin — without a source, it’s a guess, not a signal.
📄 The output file: oportunidades.md
Objective: a readable, ranked markdown file that you open in the morning and scan. Each item includes the pain point, product, and source.
How to check: open the oportunidades.md and click a source—it should take you exactly to the post that generated the signal. Without that, you can’t validate or cite it.
✓ Notify
- ✓Above the threshold and new (wasn’t in yesterday’s report)
- ✓Recurring pain across multiple sources
- ✓There’s a clickable source you can check
✗ Silence (noise)
- ✗Score below the threshold or real_pain:false
- ✗Duplicate of something already reported
- ✗Venting with no viable product behind it
🔁 Running continuously
Combine the three steps — collection, scoring, ranking — in one script and schedule it with cron (Track 4.2). The full pipeline flows from left to right: the sources feed collection, the Local LLM scores, the ranking filters and the output is your report for the day.
Read from left to right: the flow (in amber) exits the permitted sources, goes through three processing stages (in cyan) and ends at the report for the day. O cron down below closes the loop — since it’s local, repeating it all day only costs electricity.
⌨️ Copy-run main: vigia.sh
Objective: the entire pipeline in one script — collects, scores with the local model, ranks, and saves the dated markdown file.
How to check: run bash vigia.sh once manually — it should print "Recorded:" and create the oportunidades-AAAA-MM-DD.md. Only then add the cron. Keep an eye on the vigia.log in the first rounds.
Self-check (optional): why start collecting from an official source (RSS/JSON/API) instead of scraping the HTML at all costs?
📌 Module summary
Next module:
4.5 — Your local buddy with a name, face, and voice