🔮 OpenRouter — LLM Gateway
OpenRouter is INTELECTO's default provider solution. One API key, 100+ models, unified billing and automatic fallback. You switch models by changing a string.
📌 Models Available on OpenRouter
- •OpenAI’s GPT-4o and GPT-4o-mini—for general tasks
- •Anthropic's Claude 3.5 Sonnet — for reasoning and code
- •Mistral Large and Mistral 7B — excellent value for money
- •Llama 3.1 70B and 8B — open source, uncensored
- •Gemini 1.5 Pro — 1M-token context
💡 Practical Tip
Always set MODEL_NAME as an environment variable. Changing the model shouldn’t require a restart — INTELECTO reads it from .env at runtime.
🏠 Ollama — Zero Local Cost
Ollama runs LLMs directly on your hardware. Zero cost per token, complete privacy and zero network latency. The choice for sensitive data and heavy use.
📌 Recommended Models for Ollama
- •Llama 3.1 8B: fast, 8GB RAM is enough, general-purpose
- •Mistral 7B: excellent for code, lightweight for modern hardware
- •Gemma 2 9B: a good balance between reasoning and speed
- •Phi-3 Mini: tiny, runs reasonably well on a CPU
💡 Practical Tip
Ollama is ideal for development and sensitive data. For production with many users, OpenRouter is more scalable.
🧩 BaseProvider — Abstract Interface
BaseProvider is the contract every LLM provider must implement. A single method: async chat(messages) -> str. The Agent never knows which provider it is using.
📌 Why a Clean Interface Matters
- •Agent.process() calls self.provider.chat() without knowing the provider
- •Switching from OpenRouter to Ollama = changing one line of configuration
- •Testing with a mock provider = zero API cost during development
- •Create a new provider = 20 lines implementing BaseProvider
💡 Practical Tip
Use MockProvider during tests — a BaseProvider implementation that returns predefined responses. Zero cost, instant execution.
🔄 Multi-Provider Failover
ProviderChain implements automatic failover. If the primary provider fails, the secondary takes over transparently. The user doesn't notice the switch.
📌 Failover Configuration
- •PRIMARY: openrouter/anthropic/claude-3.5-sonnet
- •FALLBACK_1: openrouter/openai/gpt-4o
- •FALLBACK_2: ollama/llama3.1:8b (local)
- •Circuit breaker: disables a provider after >3 failures in 5min
- •Health check: restores the provider after 5 min of recovery
💡 Practical Tip
Configure local Ollama as the last fallback. It’s slower, but ensures Jarvis works even without internet.
🎯 Task-Based Model Selection
Different tasks have different requirements. Intelligent routing uses the right model for each task, reducing cost without sacrificing quality.
📌 Task Mapping
- •Complex code → Claude 3.5 Sonnet (better technical reasoning)
- •Simple questions → Mistral 7B (10x lower cost)
- •Long text analysis → Gemini 1.5 Pro (1M-token context)
- •Creative generation → GPT-4o (best versatility)
- •Offline tasks → Ollama llama3.1:8b (zero cost)
💡 Practical Tip
Implement the router as a function using a dictionary: task_type -> model_name. Easy to update when new models arrive.
📊 Cost Monitoring
Without cost monitoring, a misconfigured API can generate unexpected bills. cost_tracker.py tracks every token consumed.
📌 Tracked Metrics
- •Input + output tokens per request
- •Cost in USD per request (based on the model)
- •Accumulated cost for the day, week, and month
- •Top 5 most expensive tools
- •Alert when daily cost exceeds a configurable threshold
💡 Practical Tip
Set a daily cost threshold such as COST_LIMIT_USD=5.0. If it’s exceeded, Jarvis stops accepting new requests and sends an alert.
✅ Module 4.1 Summary
Next:
4.2 Skills and Tools