TL;DR

  • Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not.
  • Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%.
  • Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex.
  • Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs.
  • Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive.

The Problem: Autonomy Without a Leash

Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill.

Not because Claude was malicious. Because a background job spiraled: one failed upgrade attempt triggered a retry, which spawned a debugging session, which spawned another agent to investigate, and by the time I noticed there were 3 million tokens burned on a single broken RedisGraph upgrade.

The lesson: autonomy without guardrails is just expensive chaos. And the guardrails have to be deterministic—regex, allowlists, hard circuit breakers—not “please don’t do this” in the system prompt.

Layer 1: Deterministic Guardrails (Runs Before the LLM)

The foundation is a set of PreToolUse hooks that block dangerous commands before the LLM ever sees them. No judgment, no negotiation—if the command matches the blocklist, it’s rejected.

A few real examples:

BLOCKED: kubectl delete namespace production
BLOCKED: terraform destroy -auto-approve
BLOCKED: git push --force origin main
BLOCKED: DROP TABLE users; DROP DATABASE accounting;
BLOCKED: rm -rf /

The check runs as a regex on the command before it hits the runner:

^(kubectl.*delete|terraform.*destroy|git.*--force.*main|drop\s+(table|database)|rm.*-rf)

The agent can’t retry the command. It gets a hard exit 2 with an explanation logged. If the job requires that command (e.g., a legitimate disaster recovery terraform destroy), it either has an explicit allowlist entry or it requires human approval in a pull request—no automation.

This isn’t fancy. It’s boring and repetitive. And it works because it’s impossible to bypass.

Layer 2: Goal-Drift Detector (Weekly Embedding Cosine)

An autonomous job starts with a goal: “upgrade the monitoring stack from Prometheus 2.46 to 2.48.” Six hours later it’s committed code to add a new alerting rule, refactored three different files, and added Grafana dashboard provisioning. Related? Sure. Same goal? Not really.

The goal-drift detector runs weekly. It:

  1. Pulls the initial objective from the Langfuse trace (or the job’s README)
  2. Embeds the objective and the final commit message(s)
  3. Computes cosine similarity
  4. Alerts if the similarity is below 0.6 (roughly: “these aren’t talking about the same thing”)

A typical drift alert reads:

Objective: upgrade redis client driver
Final work: redis client driver + new caching layer + performance rewrite
Cosine: 0.42 — DRIFT DETECTED

This catches scope creep by the agent itself, not because it’s malicious, but because one fix led to another lead to another and suddenly the job did ten times the work. The alert gives me a chance to review the commits and either approve the scope expansion or revert and retry with tighter constraints.

Layer 3: Test-Deletion Sentinel (CI Regex + Hard Fail)

If an automated agent deletes a test, the CI pipeline should reject it. Not warn. Reject.

The sentinel runs as an early CI check on every MR pipeline. It looks for test file deletions in the diff:

git diff HEAD~1 | grep '^-' | grep -E '\.(test|spec)\.(js|py|go|rs)$'

If it finds a deletion, it checks for a magic tag in the commit message: [test-refactor]. If the tag exists, it also requires an issue reference (#123). If either is missing, the pipeline exits 1 and the MR is blocked. Red CI stops the merge.

This is aggressive—sometimes you do delete an old test that’s redundant—but the friction is intentional. It forces you to think about whether the test was load-bearing, and it prevents an agent from deleting tests to make a broken feature pass.

Layer 4: CronJob Failure Alert (Prometheus + Staleness)

Every background job is a Kubernetes CronJob. The kube-state-metrics exporter gives us kube_job_failed, and a simple Prometheus alert fires when:

kube_job_failed{job="background-automation"} > 0 for 2 consecutive runs

But “2 consecutive runs” isn’t enough. I also track:

  • Staleness: last successful run was >7 days ago
  • Never succeeded: job created >26 hours ago with 0 successes

This catches the insidious case where a job is failing silently and nobody notices until it’s been broken for a week. The alert goes to Slack and my monitoring dashboard, and I usually fix it the same day.

Layer 5: LiteLLM Cost-Spike Alert (24h Rolling Window)

My automation budget is ~$1.50/day for background jobs. (Cheap because I’m using Haiku for most work, not Opus.) If a job goes rogue, I want to know within 5 minutes, not after the bill arrives.

Two thresholds:

  1. 80% of daily cap (~$1.20): warning alert, I review logs
  2. 100% of daily cap (~$1.50): critical alert, new LLM requests are rejected server-side

The check is a 5-minute Prometheus scrape on cloud-model API meters. Once the cap is hit, every subsequent LLM call in the automation stack exits with “budget exhausted” and returns to the human via Slack. The job doesn’t retry. It doesn’t cascade. It stops.

Combined with the circuit breaker (see below), this prevents a runaway loop from burning $1000 in an hour.

Layer 6: Circuit Breaker (3 Failures = Breaker Open)

Before an LLM CI job runs, a pre-job check looks at the last 5 runs:

SELECT COUNT(*) as failures FROM job_runs 
WHERE job_id = $this_job AND success = false AND timestamp > now() - 5 runs

If 3 or more of the last 5 runs failed, the circuit breaker opens: the job skips the LLM call and exits 0. No retry. No cascade. A human-friendly log message goes to stderr. The next successful run closes the breaker.

This is deliberately crude—it’s not about predictive maintenance or adaptive thresholds—it’s about “if this thing keeps breaking, stop doing it until a human looks.”

Layer 7: Kill Switch (Single Env Var)

One environment variable, set in the cluster:

LLM_AUTOMATION_DISABLED=true

Every background job checks this at entry:

if os.getenv("LLM_AUTOMATION_DISABLED"):
    log("Automation disabled; exiting")
    sys.exit(0)

If the cluster is on fire and I need to stop all LLM automation immediately, I edit the ConfigMap and redeploy. All jobs respect it by the next run. No emergency credentials, no API key rotation, just a toggle.

Layer 8: Action Audit (Weekly Trace Export + Divergence Flag)

Every Friday, a job exports all LLM traces from the observability platform (Langfuse-style). It flags runs where the number of tool calls diverged wildly from the plan:

  • Plan said: “2 commits”
  • Actual: 12 diffs + 8 commits + 3 CI retries

This isn’t a blocker—sometimes plans change—but it’s a signal that something went sideways. I review these manually once a week. Usually it’s fine. Sometimes it reveals an agent that’s thrashing and needs better constraints.

Layer 9: Monitor the Monitor (Alerting on Alerts)

Here’s the thing nobody talks about: if you set up 8 different monitors, you now have 8 different alerts to check. This is alert fatigue—exactly what we’re trying to avoid.

My operating rule is: alerts are actionable or silenced. There is no third category.

Every alert I set up has a threshold that’s been tuned such that I see it maybe once a week. If I’m seeing the same alert twice a day, the threshold is wrong; I dial it up or remove the alert entirely. Same with “informational” alerts—they’re noise; I don’t run those.

The cost-spike alert, for example, only fires at 80%+ because background jobs are legitimately bursty. If I set it at 50%, I’d be looking at it constantly and it stops meaning anything.

The Fail-OPEN Principle

When a guard trips—breaker opens, kill switch hits, cost cap is reached—the job must fail open, not closed:

  • Exit 0 (success)
  • Log to stderr
  • Send a message to the human-facing chat

This trains the team to look at the chat alert, not the red CI. Red CI gets normalized; chat alerts stay novel. If I train myself to ignore red CI because it’s “just the breaker opening,” I’ll start ignoring actual failures.

What I Don’t Monitor

There are things I could monitor that I don’t:

  • Token count per run (too noisy)
  • Time-to-first-tool-call (rarely actionable)
  • “Is the LLM making progress?” (too hard to define)
  • Embedding similarity on every commit (overkill; weekly is enough)

The principle: if you can’t act on the alert, don’t fire it. Monitoring debt is real.

Lessons Learned

  1. Deterministic beats probabilistic. A regex-based blocker is worth more than a carefully-tuned prompt.
  2. Layers matter. No single guard is foolproof. A circuit breaker stops a thrashing job; a cost cap stops it even faster; a kill switch stops everything instantly.
  3. Thresholds are weapons. Tune them too tight and you’re babysitting. Too loose and you don’t see failures. I spent a month dialing these in.
  4. Humans in the loop doesn’t mean “ask permission for everything.” It means the automation goes, the monitors watch, and if something weird happens, a human notices and decides.

The cluster doesn’t babysit itself. But with these layers in place, it’s close.

Next up: how I automated dependency upgrades without burning the place down—and why Renovate’s constraints won’t save you.