TL;DR

  • I count six classifiers in my daily stack. Only three of them involve a model. The other three are keyword lists and parsers, and they are the ones I trust more.
  • The useful question for any classifier is not “how accurate is it” but “which way does it fail?” Every one of mine fails toward a human, or toward the cautious side, on purpose.
  • Four principles I keep relearning: deterministic checks first, fail toward a human, a classifier may only tighten, and measure against a golden set instead of vibes.
  • The most common classifier bug in my setup was classifying text instead of the thing the text describes.
  • Measurement is uneven. I have real numbers for some of these and none for others, and I have tried to say which is which.

Why a field guide

I did not set out to build a classifier collection. They accumulated, one incident at a time, each time something needed to decide “is this one of those” before an action happened. Resale is the hobby that funds the hobby, and a lot of the automation around it is the same question asked in different places: should this thing be allowed to proceed?

Here are the six, roughly from “someone else’s model” to “a for-loop I wrote.”

1. The permission classifier under Claude Code

When Claude Code runs in auto mode, any tool call that no allow rule covers is judged by a classifier. It is Anthropic’s, not mine, and I do not get to tune the model. What I get to tune is its context: a block of settings text that tells it which hosts, networks and accounts are mine, plus a list of allow entries.

Its job is hard and I am mostly sympathetic. It has to distinguish “agent wants to merge a branch it wrote itself” from “agent wants to merge a branch it was told to merge.” Over about seven weeks of retained transcripts I counted a few hundred denials against tens of thousands of tool calls, roughly one-half of one percent. About four in five were Bash, and nearly all were the “denied” kind, not the “could not determine safety” kind. Some were correct, such as an agent trying to merge its own merge request. Some were plain friction.

What I learned there is in the earlier post on those denials, so I will not repeat it, except for the structural point: narrow allow rules are evaluated before the classifier, so they cut classifier calls. The lever against a flaky classifier is not arguing with it. It is giving it fewer decisions. Which leads directly to the next one.

2. The destructive-command guard

Allow rules that skip the classifier also skip the safety it provides. A permissive kubectl rule means kubectl delete namespace never reaches anything that can say no. And sessions running in bypass mode have no classifier at all. So I put a deterministic hook in front of Bash: a pre-tool-use script that decides whether a command is destructive.

Version 1 was regex over the command string. It blocked terraform destroy, kubectl delete on the things that matter, force-pushes to the main branch, DROP TABLE. It also blocked a commit message that said “drop table”, a grep for a destructive pattern, and a heredoc of prose. I measured it later: replaying about 90 historic blocks, roughly 40 percent were text that mentioned a destructive command, not a destructive command. Version 1 was classifying words.

Version 2 tokenizes the shell and only applies rules where a program actually runs: the argv after sudo, inside bash -c, in an SSH remote command, in $(...), in a heredoc piped to a shell. The details are in the v2 write-up. The relevant numbers: everything v1 correctly caught was still caught, v2 found real misses where flags were in an unexpected order, and replaying about 82,000 historic commands produced zero parser exceptions.

It also has an escape path, which matters as much as the rules: an override prefix with a stated reason, accepted on the flagged command, refused in bypass mode, and logged. A guard without an escape path gets disabled the first night it is wrong.

3. The out-of-scope lot filter

Surplus auctions list a lot of things I will never flip: dump trucks, excavators, real estate, pallets of safety goggles. Evaluating those wasted effort, so a classifier decides, before anything expensive runs, whether a lot is out of scope. Matching lots get a terminal verdict and are never read, never comped and never retried.

This is the one people assume is an AI model. It is not. It is rules: a list of category substrings, a list of title keywords matched on word boundaries, a leading model year plus a vehicle make, and a few unambiguous make-and-model phrases. The last few are gated by a “tech hint” check, so “2018 MacBook Pro” never gets caught by the year rule.

The design rule is written in its docstring (trimmed here) and it is the right one:

"""Deliberately conservative: it matches strong, unambiguous signals only.
... a false skip is a missed opportunity, which is worse than one wasted
NO_COMPS cycle. When in doubt, let it through to the real evaluator."""

The cost asymmetry is explicit. A false negative (a truck gets through) costs me one wasted evaluation. A false positive (a good lot gets dropped) costs me a lot I never see. So it fails toward letting things through. The leaks I found in production were vehicles that slipped past (a category named something unexpected) and non-US lots, and each fix added a rule, never loosened one.

For where it sits in the pipeline, see the funnel post and the identification write-up.

4. The reply-status classifier

When a seller or buyer emails back, I want the contact’s deal status to reflect what they said without a human reading every message. So an inbound message is classified at capture time:

def classify_status(subject: str | None, body: str | None) -> str | None:
    """Map an inbound email to the deal status it implies, or None to leave the
    status untouched. Decline -> 'lost'; clear engagement -> 'negotiating'."""
    text = f"{subject or ''}\n{body or ''}".lower()
    if any(p in text for p in _DECLINE_PATTERNS):
        return "lost"
    if any(p in text for p in _INTEREST_PATTERNS):
        return "negotiating"
    return None

Two choices to notice. Declines are checked first, because “not interested” contains the word “interested”. And the third return value is the important one: None means “I do not know, leave it alone.” A classifier with only two outputs has to be wrong sometimes. Giving it a third, “no opinion”, lets it stay out of the way.

I have not measured its accuracy. It is a keyword list and it is probably fine, and “probably fine” is a statement about my mail, not a measurement. The consequence of being wrong is a status label on a contact that I can see and change, which is why I have not felt the need to measure it. That is a deliberate low-stakes call, and I would not make it for the lot filter’s cousin that touches money.

5. The goal judge

Claude Code’s goal feature keeps a session working until a condition is met. After each turn, a small, fast model (Haiku) reads the transcript and answers one question: is the condition met?

The catch is in what it can see. It cannot run commands or read files. It sees the transcript, only the transcript. So a condition like “all tests pass” is unverifiable unless the tests ran in the session and printed their result where the judge can see it. The patterns that work are ones where the condition is checkable from printed output (“the type-check command reports zero errors”) and the session is required to paste the evidence. Vague conditions make it oscillate, and invisible completion makes it either stop early or never stop.

I wrote that up as a skill for drafting these conditions. I do not have an accuracy figure for the judge, and I will not invent one. The practical rule is that a classifier can only judge what is in front of it, so you design the evidence to be in front of it.

6. The LLM merge-request reviewer

The last one is a model in CI that reviews merge requests. It makes one call per run, returns JSON split into blocking findings and suggestions, and posts a comment only if it found something. The details are in the CI post.

The part relevant here is ordering: deterministic policy checks run before the model sees the diff. Deleted tests, committed secrets, missing manifest validity are all regex or tool checks that cannot be talked out of a verdict. The model is the second layer, for the things regex cannot express. It also discloses its own coverage: a cap on posted findings and a note when part of the diff was not seen, so a quiet review never silently means “I did not look.”

The principles

Deterministic first. If a rule can decide it, a rule should. Rules are testable, fast, free and do not drift when a model is updated. Three of my six are entirely rules, and two of the three that are not (the permission gate and the reviewer) have rules running in front of them.

Fail toward a human. For each classifier, I decided what an error does. The lot filter errs toward evaluating. The guard, when it cannot parse a command, falls back to the stricter text rules. The reply sorter returns “no opinion”. The reviewer discloses what it skipped. None of them default to “silently proceed on an uncertain call.”

A classifier may only tighten. This is the one I hold hardest. In the lot pipeline, a final sanitizer can cap a bid or demote a verdict, but there is no code path where it makes a verdict more aggressive. The lot filter can only remove things from consideration. Anything that can loosen a restriction on its own judgment will, eventually, loosen the wrong one. When something does need to relax, a human does it.

Measure against a golden set. Opinions about classifiers are cheap. A small set of cases with known answers, run before and after every change, is not. I do this seriously for the vision side, and the method is in the golden-set bench post. For the keyword classifiers, I have only partial measurement. The command guard has a replay corpus, and the rest rely on the cost of being wrong being low. That is the honest state of things, and it is a to-do list.

The bug class that kept recurring

If I had to name one failure across all six, it is classifying the description instead of the thing. The v1 guard blocked a commit message about a destructive command. The lot filter’s year rule would have caught a laptop whose title began with a model year. The reply sorter reads “not interested” and nearly read “interested.” The goal judge trusts a sentence claiming the tests passed unless it can see the output.

In each case the fix was the same: move the decision closer to the real object. Parse the argv, not the string. Require a tech hint before applying the year rule. Check the decline first. Make the evidence appear in the transcript.

Where I would go next

I would like a number for the reply sorter, which means hand-labelling a few hundred messages, which I have not done. And I would like the permission-classifier friction to be someone else’s problem, which it mostly is, except when it is not.

If you build one of these, write down which way it fails before you write the rules. The rest follows from that.