Operations

TL;DR Five rungs: Rung 0 (read-only) → Rung 1 (draft MRs) → Rung 2 (commit on /llm fix) → Rung 3 (auto-merge patches) → Rung 4 (exceptions-only). Promotion is earned, not promised — ~30 days at Rung 1 + <10% reverts → Rung 2; ~60 days at Rung 2 + <5% reverts → Rung 3. Infra agents cap at Rung 2 — Kubernetes, Terraform, secrets, cert-manager always need human approval to merge; blast radius is too high. Demotion is immediate on test deletion, secret leak, scope creep, or any deploy needing manual rollback. Non-negotiable gates (deterministic, run at the CI runner) block everything: no deleted tests, no secrets, no medium+ CVEs, destructive commands on allowlist only. From concept to operational I wrote about the trust ladder for agents in May — the philosophy of supervised → monitored → trusted → autonomous, and why you can’t just flip a switch. That post is the why; this one is the how. I’m going to walk through the exact rungs, the promotion criteria that actually work, the demotion triggers, and the CI rules that make it safe. ...

TL;DR Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not. Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%. Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex. Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs. Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive. The Problem: Autonomy Without a Leash Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill. ...

TL;DR This is the counterpart to the manifesto and the DR drill. After moving a chunk of the stack home, a list of things deliberately stayed rented: Route53, ACM, S3, AWS KMS, the Anthropic API for Claude, Bedrock for Amazon-only models, a transactional email sender, and one repo on GitHub. Each of them earns its place by being either the long pole on availability or the dependency that has to outlive the cluster. Self-hosting maximalism is a trap; the seam is the feature. ...

TL;DR A year ago my stack was the usual mix — GitHub for code, ECR for images, GitHub Actions for CI, Docker Hub for upstreams, Route53 + S3 + CloudFront for the blog. Most of that’s still where it should be. About a third of it isn’t. This post is the retrospective on what came home, what stayed rented, and the rule of thumb I now use when deciding which side of the line a new service goes on. The short version: self-host the things you operate; rent the things you’d never have time to operate. ...

TL;DR A homelab feels free until you read the meter. After a year of running seven k3s nodes plus a pair of Mac Studios under whatever workload I felt like throwing at them, I sat down with a Kill-a-Watt and worked out what the cluster actually costs to keep on. Idle is genuinely cheap. Sustained LLM inference is not. The honest break-even against cloud inference is workload-shaped, and for my workloads, on-prem wins — but only because I run them often enough to amortize the wattage. The numbers below are mine; substitute your electricity rate to get yours. ...

TL;DR I run a growing fleet of autonomous agents — homelab ops, trading research, content generation. Most blow up the first few times they try anything new. I needed a way to decide what an agent is allowed to do without asking me, and what still requires a human checkpoint. The answer is a four-rung trust ladder — supervised, monitored, trusted, full autonomy. Agents earn rungs through track record, not promises. Demotions are possible and routine. The framework took the question “should this agent be allowed to do X” out of my head every single time and turned it into a policy I can apply consistently. ...