Observability

TL;DR I run unattended LLM agents on my homelab — they write code, open MRs, generate content, rotate secrets. The problem: they fail silently and bill silently. Langfuse (a tracing platform) logs every LLM call with input/output tokens, latency, and cost. On top of those traces, I built three background monitors that run weekly: a goal-drift detector that compares an agent’s stated objective to what its commits actually did (via embedding similarity), a cost-spike alert that fires at 80% and 100% of a daily budget cap, and an action audit that exports traces and flags sessions where the tool-call sequence diverged from the plan. Together, these let me sleep while autonomous agents handle repetitive work. ...

TL;DR A new Opus model shipped, so I sat down to re-tune the agent harness I drive it with — the CLAUDE.md files, skills, hooks, and settings that shape every session. The surprising part: the most valuable changes weren’t trimming prompts for the smarter model. They were wiring the agent into infrastructure I already run — offloading bulk work to a local LLM (≈$0), a live homelab statusline, session tracing for an “action audit,” and a goal-drift monitor that uses the local model as judge. I also learned not to trust the new model’s own suggestions about what to cut. It wanted to delete load-bearing guardrails. ...

TL;DR My Mac Studio M3 Ultra runs Ollama with 70B+ models but isn’t a k3s node. I needed it to show up in Grafana next to the cluster workloads. The solution: node_exporter for system metrics, a Go reverse proxy for per-model inference metrics, a custom Python exporter for model inventory and VRAM tracking, and Grafana Alloy for shipping logs to Loki. All four services managed by Ansible, all metrics scraped by the cluster’s Prometheus. ...