Homelab

TL;DR Secrets leak through echo, debug tracing, logs, process args, and env-var inheritance — guard the flow, not just storage Fetch from Bitwarden once, assign to a shell variable, reference the variable — never inline the raw value Verify with first-3/last-3: ${VAR:0:3}...${VAR: -3} — confirm you have the right token without exposing it GitLab masked vars hide values in logs (8+ chars); protected vars only expose on protected branches — use both for production secrets Agent permission rules prefer command substitution inside the allowed command, not as an env var Rotate tokens immediately; verify before use; pin your bw CLI version The leak surface — where secrets escape I used to think storing secrets in Bitwarden (or Vault, HashiCorp, whatever) meant they were safe. I was wrong. ...

TL;DR Three production MCP servers I run: a docs/wiki wrapper, a personal-finance aggregator, and an inventory-ops tool for a side project. Type hints are your schema. The FastMCP SDK extracts JSON Schema from Python docstrings and type annotations — you almost never hand-write schemas. Flatten nested APIs. If the upstream API returns nested JSON, unwrap it in the MCP tool and return clean markdown or a simple dict — agents prefer predictable structures. Reads always-on, writes gated. Use environment variables (MCP_WRITE=1) or similar to gate mutation tools; put mutations behind a validated service layer and audit every call. Log to stderr, not stdout. Stdio protocol uses stdout for JSON-RPC; anything else breaks the connection. The lineup I’ve written three MCP servers from scratch over the last few months. This is the cookbook — concrete patterns, not theory. If you haven’t read the intro to writing MCP servers, start there; I won’t re-explain the anatomy. ...

TL;DR Five rungs: Rung 0 (read-only) → Rung 1 (draft MRs) → Rung 2 (commit on /llm fix) → Rung 3 (auto-merge patches) → Rung 4 (exceptions-only). Promotion is earned, not promised — ~30 days at Rung 1 + <10% reverts → Rung 2; ~60 days at Rung 2 + <5% reverts → Rung 3. Infra agents cap at Rung 2 — Kubernetes, Terraform, secrets, cert-manager always need human approval to merge; blast radius is too high. Demotion is immediate on test deletion, secret leak, scope creep, or any deploy needing manual rollback. Non-negotiable gates (deterministic, run at the CI runner) block everything: no deleted tests, no secrets, no medium+ CVEs, destructive commands on allowlist only. From concept to operational I wrote about the trust ladder for agents in May — the philosophy of supervised → monitored → trusted → autonomous, and why you can’t just flip a switch. That post is the why; this one is the how. I’m going to walk through the exact rungs, the promotion criteria that actually work, the demotion triggers, and the CI rules that make it safe. ...

TL;DR Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not. Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%. Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex. Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs. Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive. The Problem: Autonomy Without a Leash Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill. ...

TL;DR I ran an audit over my repos going back to March: pull the first commit that introduced each agentic pattern, then go find out when that pattern actually became common practice in the wider ecosystem. The results were humbling in a useful way. Four things I quietly considered myself early on were catch-up, in one case by two years. Four other things turned out to be genuinely ahead of documented practice. The interesting part is what those four have in common, and it is not what I expected. ...

TL;DR One model = one blind spot. Claude and Gemini were trained on different datasets, have different architectures, and will miss different things. Consult for breadth, not consensus. The goal is to widen your analysis surface and catch blindspots, not to reach agreement. If all three models agree, that’s suspicious—run stress tests. The bridge is simple: Playwright drives a logged-in Gemini tab via Chrome DevTools Protocol, pastes your prompt, and scrapes the response when done. Real session, no API key. Two modes: (1) cold second-opinion—hand off a problem and ask for an independent take; (2) adversarial debate—assign sides, force defense, then reconcile. Both work; adversarial is brutal and useful for high-stakes calls. Anchoring is the trap. If you show Model B what Model A said and ask “do you agree?”, you get theater. Always pose cold or ask for the opposite argument first. Every Model Has Its Blind Spot I make the same mistakes in code as everyone else. My reasoning hits walls. I miss architectural gotchas and cut corners on testing. An LLM does the same, just in different places. ...

TL;DR Two Claude agents running in separate project contexts share a private Mattermost channel to ask each other for help without human intervention. A deterministic polling gate (bash + curl) checks for new messages before spawning the expensive responder LLM — idle polls cost almost nothing. The responder runs on the Claude Code subscription (keychain OAuth), not API credits, and keeps the LLM cheap. Scheduled every ~3 minutes by macOS launchd; armed/disarmed by loading/unloading the launch agent. Supervised autonomy: the responder can read, triage, draft, and explain — but NOT commit, spend money, change access, or deploy. Why one agent asks another I run parallel Claude Code sessions on different projects. They’re isolated — each has its own repo context, its own workflow, its own focus. But sometimes they need each other. ...

TL;DR I point Claude at a problem, give it a direction, and let it run — and most of the time I don’t read the merge request it opens. On an internal homelab where the blast radius is my own cluster, backups are real, and everything’s reproducible from code, line-by-line diff review is the wrong altitude. Instead I supervise after the fact by reading the artifacts the agents leave behind: the /docs/ folder, a Wiki.js wiki full of Mermaid diagrams, auto-generated architecture SVGs, and this blog. Reading those — not the diffs — is what’s actually caught problems: dead systems still wired in, duplicate config, a deploy that quietly deleted itself. ...

TL;DR The headless problem: Running a refurb bench means stacking PCs, but monitors/keyboards create noise and clutter. You need remote hands. PiKVM V4 Plus is a network KVM: remote video capture, USB keyboard/mouse emulation, virtual media mounting, and ATX relay control—all in a $300 device. Multiport Switch extender lets one PiKVM control up to four machines simultaneously by flipping an RJ45 port. Dell CCTK (Command | Configure) automates BIOS settings from WinPE: set AHCI, enable PXE, disable Secure Boot, configure Wake-on-LAN, tag assets—all without touching F2. Unattended imaging = BIOS automation + PXE boot + virtual media mounting. Chain ten machines without hands-on intervention. Gotchas matter: AHCI after Windows install = BSOD; Secure Boot changes need a setup password; BitLocker auto-encrypts Win11 and blocks generalize; Deep Sleep disables S5 wake. Why I Needed This I’ve been buying government-surplus refurbished machines to flip—mostly surplus OptiPlex machines—and my original setup was a single monitor, keyboard, and mouse swapped between machines. That works for one box. It doesn’t scale to five. ...

TL;DR Cover art was the gateway drug. The same local ComfyUI install that generates this blog’s headers also strips the cluttered background off a photo of hardware on my bench, upscales a small generation to retina resolution, and batch-produces a consistent set of illustrations from a prompt template. Two Mac Studios mean I can fire a batch at one box and keep working on the other. It’s all driven from scripts and agents, and it all costs $0 per image because it never leaves the house. ...