A set of classifiers each routing decisions toward allow, deny or a human A set of classifiers each routing decisions toward allow, deny or a human

A field guide to the classifiers in my stack

TL;DR I count six classifiers in my daily stack. Only three of them involve a model. The other three are keyword lists and parsers, and they are the ones I trust more. The useful question for any classifier is not “how accurate is it” but “which way does it fail?” Every one of mine fails toward a human, or toward the cautious side, on purpose. Four principles I keep relearning: deterministic checks first, fail toward a human, a classifier may only tighten, and measure against a golden set instead of vibes. The most common classifier bug in my setup was classifying text instead of the thing the text describes. Measurement is uneven. I have real numbers for some of these and none for others, and I have tried to say which is which. Why a field guide I did not set out to build a classifier collection. They accumulated, one incident at a time, each time something needed to decide “is this one of those” before an action happened. Resale is the hobby that funds the hobby, and a lot of the automation around it is the same question asked in different places: should this thing be allowed to proceed? ...

October 22, 2026 · 10 min · zolty
A small golden set of lots scored with paired comparisons and confidence intervals A small golden set of lots scored with paired comparisons and confidence intervals

Benching a vision pipeline on 17 lots without fooling myself

TL;DR My pipeline reads photos of surplus lots, finds comparable sales, and does arithmetic to produce a bid ceiling. To change any part of it safely, I need a measurement that can tell a real improvement from luck. The method: a small golden set of lots with human-verified answers, paired comparisons (every candidate sees the same lots), an exact McNemar test on the cases where candidates disagree, and Wilson intervals on every rate. On 17 lots, “accuracy %” is noise. A single lot moves the number by about six points, and the 95 percent interval on a respectable-looking score is roughly thirty points wide. Reading make and model is the easy part. Counting units and choosing comps is where the error enters, so I score those separately from identification. The measurement that actually changed my mind was not a benchmark at all. It was re-running a batch of lab lots by hand: the summed bid ceilings dropped by about 72 percent. No per-model leaderboard here, on purpose. The method is the transferable part, and a small bench makes model rankings mostly an artifact. This is the method behind the numbers in the identification post. Where that post shows how the pipeline works, this one is about how I know whether a change made it better. ...

October 19, 2026 · 9 min · zolty
Reading option logits from a small local model Reading option logits from a small local model

Jev in 25 lines: building the decision model locally

TL;DR In the design review I concluded that Jev, TypeSafe AI’s typed decision model, is cheaper and faster than an LLM and not more accurate. That raised an obvious build-vs-buy question: if the value is “cheap probability over a fixed set of options,” can I get that from a small open model running on hardware I already own? Mostly yes, with one catch that matters. The technique is to prompt with lettered options, run one forward pass, and read the logits for just the option letters. A parody post called “Jev in 25 lines of Python” does exactly this, and the technique under the joke is real. The catches are calibration (raw softmax is bad until you fit it) and serving (Ollama could not do it when I wrote the design). ...

October 16, 2026 · 8 min · zolty
A gallery of surplus lot photos feeding a structured identification schema A gallery of surplus lot photos feeding a structured identification schema

Reading a surplus lot from its photos: the identification pipeline

TL;DR Resale is the hobby that funds the hobby, and the expensive mistake in it is not a bad price. It is a confident wrong model number, because everything downstream (comps, margin, bid ceiling) is built on it. Auction search results give you one thumbnail and a title the seller typed in a hurry. The real information is in the gallery: wide shots for counting, close-ups for labels. The pipeline is split on purpose: vision reads the lot, a comps step prices what was read, and plain deterministic arithmetic turns that into a number. The model never does the math. Fetching the full gallery moved make/model accuracy from 0.375 to 0.875 on a 24-lot hand-labelled bench. That is the single biggest lever I have found, bigger than any model swap. Most of what went wrong later was not misreading a badge. It was counting and itemizing: accessories sold as products, “X with Y” counted twice, a screen size parsed as a quantity. A final sanitizer can only tighten a verdict. And if the gallery fetch fails, the lot is deferred, not guessed. I wrote about how lots get found in the first place. This post is about the step after: given a lot that looks interesting, how do I know what is actually in the pile? ...

October 13, 2026 · 11 min · zolty
Security model evaluation results Security model evaluation results

Grading Seven AI Models on Blue-Team Security Work in a VM With No Network Card

TL;DR I wanted to know which models I can trust for the defensive side of running a homelab: reviewing code for vulnerabilities, reviewing Kubernetes and Terraform config, triaging logs, and reading threat intel without making things up. So I built a harness with 112 frozen synthetic cases, graded everything deterministically (no LLM judge), and ran every model-written fix inside a Proxmox VM whose network card was deleted at the hypervisor. ...

October 12, 2026 · 16 min · zolty
A design review of a small typed decision model A design review of a small typed decision model

Reviewing Jev: a decision model that is cheaper, not smarter

TL;DR Jev is TypeSafe AI’s first “System One” model, public since 2026-09-15. You send it a state object and a few typed questions, and it returns probabilities in 70 to 500 ms. I wrote a design doc for using it in my own tooling, and then rewrote it twice in two days because the first draft was wrong in ways I only noticed by rereading the sources. Nothing is built. There is no client, no table, no spike, no measured number of mine in this post. This is a review of a third-party product before I adopt it. Every accuracy and cost figure below belongs to the vendor or to a third party, and I label it that way. ...

September 25, 2026 · 11 min · zolty
A funnel from surplus auction listings to a ranked shortlist A funnel from surplus auction listings to a ranked shortlist

Two dozen crawlers and a shortlist: how I stopped browsing surplus auctions

TL;DR Government and institutional surplus auctions are a firehose: thousands of live lots, most of them badly photographed, vaguely titled, and worthless. Browsing them by hand doesn’t scale and — worse — it’s biased. You click the pretty photos, and the pretty photos are the ones everyone else clicked. The fix is a funnel: ~24 scheduled crawlers feeding a single Postgres table, photos and paperwork in object storage, an hourly evaluation pass, and one ranked board. The judgement layer (reading the goods out of photos, estimating what they’re worth) runs as a small model pipeline with a second pass that checks the first. I’m deliberately light on its internals here. The output isn’t a decision. It’s a shortlist with reasons attached — and a “this is wrong” button that puts the lot back in the queue. The single highest-value change in three months was raising how many photos get looked at. The label is often in photo nine. None of this is my job. It’s the hobby that pays for the hobby, which is the only reason it’s allowed to be this over-engineered. Why I built a data pipeline for a weekend habit Let’s be honest about the proportions here. I built two dozen scheduled crawlers, an object store, an hourly evaluation pass and a ranked dashboard, in order to buy second-hand equipment on the internet slightly better than I did before. ...

August 7, 2026 · 11 min · zolty
Auditing repository history against ecosystem adoption timelines Auditing repository history against ecosystem adoption timelines

I audited four months of my own repos to see what I was actually early on

TL;DR I ran an audit over my repos going back to March: pull the first commit that introduced each agentic pattern, then go find out when that pattern actually became common practice in the wider ecosystem. The results were humbling in a useful way. Four things I quietly considered myself early on were catch-up, in one case by two years. Four other things turned out to be genuinely ahead of documented practice. The interesting part is what those four have in common, and it is not what I expected. ...

July 19, 2026 · 8 min · zolty
Routing a diff to a second vendor's model for adversarial review before merge Routing a diff to a second vendor's model for adversarial review before merge

The merge gate is a rival model: four days of Gemini deleting my code

TL;DR Before I merge anything where being wrong would be silent, I paste it into a live Gemini session and ask that model to attack it. Not “review this,” which gets you a rubber stamp with bullet points. Attack it. Over four days in April, that gate killed a performance optimization 88 minutes after it shipped, found two engine bugs that meant every historical number in my trading leaderboard was inflated, and identified five separate ways my “out-of-sample” evaluator was reading the future. Claude wrote all of that code. Claude also reviewed all of that code. Claude caught none of it. The useful property is not that Gemini is smarter, it is that Gemini did not write the thing and has no investment in the design being correct. ...

July 19, 2026 · 9 min · zolty
Multiple language models consulted as a panel Multiple language models consulted as a panel

A panel of LLMs: using Gemini and Claude to pressure-test decisions

TL;DR One model = one blind spot. Claude and Gemini were trained on different datasets, have different architectures, and will miss different things. Consult for breadth, not consensus. The goal is to widen your analysis surface and catch blindspots, not to reach agreement. If all three models agree, that’s suspicious—run stress tests. The bridge is simple: Playwright drives a logged-in Gemini tab via Chrome DevTools Protocol, pastes your prompt, and scrapes the response when done. Real session, no API key. Two modes: (1) cold second-opinion—hand off a problem and ask for an independent take; (2) adversarial debate—assign sides, force defense, then reconcile. Both work; adversarial is brutal and useful for high-stakes calls. Anchoring is the trap. If you show Model B what Model A said and ask “do you agree?”, you get theater. Always pose cold or ask for the opposite argument first. Every Model Has Its Blind Spot I make the same mistakes in code as everyone else. My reasoning hits walls. I miss architectural gotchas and cut corners on testing. An LLM does the same, just in different places. ...

July 17, 2026 · 8 min · zolty

Affiliate Disclosure: Some links on this site are affiliate links (Amazon Associates, DigitalOcean referral). As an Amazon Associate, I earn from qualifying purchases. This does not affect the price you pay or my editorial independence — I only recommend products and services I personally use and trust.