Operations

TL;DR Secrets leak through echo, debug tracing, logs, process args, and env-var inheritance — guard the flow, not just storage Fetch from Bitwarden once, assign to a shell variable, reference the variable — never inline the raw value Verify with first-3/last-3: ${VAR:0:3}...${VAR: -3} — confirm you have the right token without exposing it GitLab masked vars hide values in logs (8+ chars); protected vars only expose on protected branches — use both for production secrets Agent permission rules prefer command substitution inside the allowed command, not as an env var Rotate tokens immediately; verify before use; pin your bw CLI version The leak surface — where secrets escape I used to think storing secrets in Bitwarden (or Vault, HashiCorp, whatever) meant they were safe. I was wrong. ...

TL;DR Five rungs: Rung 0 (read-only) → Rung 1 (draft MRs) → Rung 2 (commit on /llm fix) → Rung 3 (auto-merge patches) → Rung 4 (exceptions-only). Promotion is earned, not promised — ~30 days at Rung 1 + <10% reverts → Rung 2; ~60 days at Rung 2 + <5% reverts → Rung 3. Infra agents cap at Rung 2 — Kubernetes, Terraform, secrets, cert-manager always need human approval to merge; blast radius is too high. Demotion is immediate on test deletion, secret leak, scope creep, or any deploy needing manual rollback. Non-negotiable gates (deterministic, run at the CI runner) block everything: no deleted tests, no secrets, no medium+ CVEs, destructive commands on allowlist only. From concept to operational I wrote about the trust ladder for agents in May — the philosophy of supervised → monitored → trusted → autonomous, and why you can’t just flip a switch. That post is the why; this one is the how. I’m going to walk through the exact rungs, the promotion criteria that actually work, the demotion triggers, and the CI rules that make it safe. ...

TL;DR Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not. Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%. Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex. Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs. Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive. The Problem: Autonomy Without a Leash Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill. ...

TL;DR I point Claude at a problem, give it a direction, and let it run — and most of the time I don’t read the merge request it opens. On an internal homelab where the blast radius is my own cluster, backups are real, and everything’s reproducible from code, line-by-line diff review is the wrong altitude. Instead I supervise after the fact by reading the artifacts the agents leave behind: the /docs/ folder, a Wiki.js wiki full of Mermaid diagrams, auto-generated architecture SVGs, and this blog. Reading those — not the diffs — is what’s actually caught problems: dead systems still wired in, duplicate config, a deploy that quietly deleted itself. ...

TL;DR I keep four small-form-factor PCs on a bench for testing and repurposing — bought used, need fresh OS images, fresh BIOS settings, and no monitor or keyboard. A PiKVM V4 Plus with a multiport switch gives me eyes and hands on all four boxes over the network. Dell’s cctk command-line tool (Command | Configure) lets me bake BIOS settings — boot order, AHCI mode, Wake-on-LAN, power-on-after-failure — into scripted runs instead of clicking through F2 menus. No monitor, no keyboard, no physical access for weeks at a time. Everything repeatable, everything as code. ...

TL;DR I’ve wired LLMs into my GitLab CI pipeline to auto-review merge requests, post findings as comments, and (on command) generate patches and commit fixes. The key insight: deterministic gates run first. Before the LLM ever sees a diff, regex-enforced checks block deleted tests, committed secrets, and destructive commands. Regex is certain; LLM judgment is probabilistic. Gate first, judge second. The bot reviews silently unless it finds something, posts to the MR with confidence levels, and can be leveled up from read-only observer to trusted committer as it proves itself — hence the “autonomy ladder” (Rungs 0–4) that gates who decides what. Infrastructure repos cap at Rung 2 (never auto-merge). ...

TL;DR I run unattended LLM agents on my homelab — they write code, open MRs, generate content, rotate secrets. The problem: they fail silently and bill silently. Langfuse (a tracing platform) logs every LLM call with input/output tokens, latency, and cost. On top of those traces, I built three background monitors that run weekly: a goal-drift detector that compares an agent’s stated objective to what its commits actually did (via embedding similarity), a cost-spike alert that fires at 80% and 100% of a daily budget cap, and an action audit that exports traces and flags sessions where the tool-call sequence diverged from the plan. Together, these let me sleep while autonomous agents handle repetitive work. ...

TL;DR I put a LiteLLM proxy gateway in front of every LLM I use — local Ollama models for bulk/cheap classification work, OpenRouter for frontier models when I need them, plus cloud vendors if needed. Every app and agent targets one OpenAI-compatible endpoint. Per-key budgets and daily spend alerts make runaway costs impossible. I define model-to-backend mappings in YAML, let LiteLLM handle the routing, and route based on intent: ask for solar-expert when I need a domain-specific Q&A bot backed by a small local model, ask for claude-opus-4-8 when I need real reasoning. The gateway cost? ~50ms latency overhead and one Kubernetes Deployment. The gain? No more vendor SDK sprawl, no more guessing which model is wired into a cron job, and spend visibility that I actually trust. ...

TL;DR Every service I run — ComfyUI, Grafana, Vault, even the ancient app on a Mac across the network — lives behind a Traefik forward-auth middleware that hands off to Authentik. No per-service login page. One Authentik login shared across everything. The magic is a two-route IngressRoute pattern: a protected route with the middleware + an unprotected callback route for the OAuth flow itself. Adding a new service to the cluster takes five lines of YAML. Wiring a non-Kubernetes backend — like the Mac that runs ComfyUI and Ollama — takes a service-with-manual-endpoints proxy. ...

TL;DR A new Opus model shipped, so I sat down to re-tune the agent harness I drive it with — the CLAUDE.md files, skills, hooks, and settings that shape every session. The surprising part: the most valuable changes weren’t trimming prompts for the smarter model. They were wiring the agent into infrastructure I already run — offloading bulk work to a local LLM (≈$0), a live homelab statusline, session tracing for an “action audit,” and a goal-drift monitor that uses the local model as judge. I also learned not to trust the new model’s own suggestions about what to cut. It wanted to delete load-bearing guardrails. ...