Langfuse

TL;DR Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not. Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%. Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex. Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs. Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive. The Problem: Autonomy Without a Leash Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill. ...

TL;DR I run unattended LLM agents on my homelab — they write code, open MRs, generate content, rotate secrets. The problem: they fail silently and bill silently. Langfuse (a tracing platform) logs every LLM call with input/output tokens, latency, and cost. On top of those traces, I built three background monitors that run weekly: a goal-drift detector that compares an agent’s stated objective to what its commits actually did (via embedding similarity), a cost-spike alert that fires at 80% and 100% of a daily budget cap, and an action audit that exports traces and flags sessions where the tool-call sequence diverged from the plan. Together, these let me sleep while autonomous agents handle repetitive work. ...