Ai-Agents

TL;DR Claude Code’s auto permission mode puts a classifier between the agent and any tool call that is not already covered by an allow rule. I ran in that mode all summer. In the transcripts I still have (mid-August to early October), 301 tool results mention that classifier. About 288 are outright denials, about ten are “no verdict” errors, and a few just quote the phrase. Of roughly 57,700 tool calls in September, that is about 0.5%. Almost all of it landed in September (285 of the 301), and it peaked on September 6 with 29 in one day. Two things helped: adding environment text that tells the classifier what my infrastructure is, and adding narrow allow rules so routine commands never reach it. Neither one was a clean win, and I will show where the data does and does not support that. Sometime between late September and early October I switched the default mode to bypass. The thing that makes that tolerable is a deterministic hook that runs before permission rules in every mode. The classifier was not wrong often. It was expensive in attention, and it was the wrong tool for unattended work. This is part of a small batch about the classifiers I run around agents. The rewrite of my destructive-command guard came first, and a field guide to all of them will follow. ...

TL;DR In July I wrote about a test-deletion sentinel: CI hard-fails if a merge request deletes tests, unless the commit message carries a [test-refactor] tag and an issue number. I was pleased with it. It turns out a tag in a commit message is a bypass that an agent (or I, on a bad day) can write in one line, and it excused every test change in the MR, not just the one I meant. ...

TL;DR I run a PreToolUse hook that blocks a short list of unrecoverable shell commands before my coding agent can run them. Version 1 was a list of regular expressions over the raw command string. Replaying 90 historical blocks and about 82,000 historical commands showed that roughly 40% of v1’s blocks were not commands at all. They were grep patterns, commit messages, and markdown heredocs that happened to contain a scary phrase. v2 tokenizes the command the way a shell would and applies each rule only where a program actually runs. On the same replay it allowed every false text match, still caught every real destructive command v1 caught, caught a few that v1 missed because of flag order, threw zero parser errors, and took about 51 ms per call end to end. It fails toward blocking. If it cannot parse something it falls back to text rules, and if Python is missing the shell wrapper falls back to the old regex list. It is a tripwire for an agent that makes mistakes, not a security boundary against one that is trying to hide. I show a case it does not catch below. I will write up my experience with Claude Code’s auto-mode classifier separately. The short version is that this hook, not the classifier, is what stands between the agent and my cluster’s worst commands. ...

TL;DR Between late June and late September I moved my daily setup through Sonnet 5, Opus 5, Fable 5.1, Opus 5.5 and Sonnet 5.5. Haiku stayed at 4.5 the whole time. I never saw a Haiku 5 in any log. There is no controlled benchmark in anything I kept. What follows is operational evidence: what I changed, what broke, what I verified, and what I would have gotten wrong if I had trusted what a model told me about itself. The changes that mattered were small: a per-model effort block in settings.json and a subagent default model. The expensive lessons were two traps. One is a bug where background-job subagents silently ran Haiku 4.5. The other is that a worker’s report of its own model cannot be believed. If you want the earlier, more general version of this, see re-tuning my setup for a new Opus model. This is what the next five releases taught me. The ladder These are first-seen dates from my own records: commit trailers in my repositories and, where I still have them, session transcripts. They are when I started using each model, not necessarily when it became available. ...

TL;DR On September 6 I audited every file Claude Code loads at the start of a session on my main machine. The global CLAUDE.md went from 19,286 bytes to 7,607 (down 60.6%). The four biggest project memory indexes went from 64,900 bytes combined to 6,513 (down 90.0%). The file had more than doubled in under three months. A backup from June 18 was 9,242 bytes. Every addition had seemed reasonable when I made it. The biggest single category of cuts was model information: release lists, retirement dates, introductory pricing, spend estimates. I replaced all of it with a rule, “use only the models and efforts the active CLI offers”, and kept the judgment about which tier fits which task. The honest caveat: this is a document check. I verified that files parse, links resolve, and the constraints I care about survived. I did not benchmark whether Claude behaves better or worse, and I did not take a token count from a live session. Bytes are not tokens. If you have read my earlier post on re-tuning my setup for a new model, this is the sequel with a different lesson. That time I learned not to trust a model’s suggestions about what to delete. This time I put a validator and a rollback around the edits. ...

TL;DR On July 31 a fleet of long-running Claude manager sessions that I was using to supervise parallel work collapsed badly enough that I had to cold-start the whole thing. My first hypothesis was that the supervisor model was not smart enough. That hypothesis was checkable, and false. The transcripts showed the managers really were running Opus 5. They still failed. The dominant failure markers in their ledgers were stale and duplicate, which are bookkeeping problems that a script gets right every time and a model gets right most of the time. The same week, a Sonnet auditor at medium effort, given an explicit list of gates to check, found two real defects that the Opus supervisor had missed. The Opus supervisor was better at something else: writing the gate list and noticing what was not on it. I turned that into a table that allocates model and effort per type of turn, with scripts doing the work that is decidable. I am going to be careful about how much that one afternoon proves, which is not much. It is one observation, and I did not run a benchmark. I wrote earlier about sending finished code to a rival model for review. This is the same instinct from the other direction: reviewer and author should not share a blind spot, but they also do not need to share a price tier. ...

TL;DR A new Opus model shipped, so I sat down to re-tune the agent harness I drive it with — the CLAUDE.md files, skills, hooks, and settings that shape every session. The surprising part: the most valuable changes weren’t trimming prompts for the smarter model. They were wiring the agent into infrastructure I already run — offloading bulk work to a local LLM (≈$0), a live homelab statusline, session tracing for an “action audit,” and a goal-drift monitor that uses the local model as judge. I also learned not to trust the new model’s own suggestions about what to cut. It wanted to delete load-bearing guardrails. ...

TL;DR AI agents fail when they don’t know what you know. I built a Slack bot that conducts structured 5-layer interviews to extract tacit knowledge — operating rhythms, decision criteria, dependencies, friction points, leverage opportunities — and generates soul.md, user.md, and heartbeat.md config files for provisioning agents. The interview surfaces ~30% more actionable context than documentation alone. Full source code below. The Problem Nobody’s Talking About Nate B. Jones has a video that nails the core issue with AI agents: they fail because they lack tacit knowledge. Not the stuff in your docs — the stuff in your head. The 20-year veteran who just knows that the staging deploy takes longer on Thursdays because the batch job runs. The designer who can feel when a color palette is wrong without being able to articulate why. ...