Evaluation

TL;DR I count six classifiers in my daily stack. Only three of them involve a model. The other three are keyword lists and parsers, and they are the ones I trust more. The useful question for any classifier is not “how accurate is it” but “which way does it fail?” Every one of mine fails toward a human, or toward the cautious side, on purpose. Four principles I keep relearning: deterministic checks first, fail toward a human, a classifier may only tighten, and measure against a golden set instead of vibes. The most common classifier bug in my setup was classifying text instead of the thing the text describes. Measurement is uneven. I have real numbers for some of these and none for others, and I have tried to say which is which. Why a field guide I did not set out to build a classifier collection. They accumulated, one incident at a time, each time something needed to decide “is this one of those” before an action happened. Resale is the hobby that funds the hobby, and a lot of the automation around it is the same question asked in different places: should this thing be allowed to proceed? ...

TL;DR My pipeline reads photos of surplus lots, finds comparable sales, and does arithmetic to produce a bid ceiling. To change any part of it safely, I need a measurement that can tell a real improvement from luck. The method: a small golden set of lots with human-verified answers, paired comparisons (every candidate sees the same lots), an exact McNemar test on the cases where candidates disagree, and Wilson intervals on every rate. On 17 lots, “accuracy %” is noise. A single lot moves the number by about six points, and the 95 percent interval on a respectable-looking score is roughly thirty points wide. Reading make and model is the easy part. Counting units and choosing comps is where the error enters, so I score those separately from identification. The measurement that actually changed my mind was not a benchmark at all. It was re-running a batch of lab lots by hand: the summed bid ceilings dropped by about 72 percent. No per-model leaderboard here, on purpose. The method is the transferable part, and a small bench makes model rankings mostly an artifact. This is the method behind the numbers in the identification post. Where that post shows how the pipeline works, this one is about how I know whether a change made it better. ...

TL;DR Resale is the hobby that funds the hobby, and the expensive mistake in it is not a bad price. It is a confident wrong model number, because everything downstream (comps, margin, bid ceiling) is built on it. Auction search results give you one thumbnail and a title the seller typed in a hurry. The real information is in the gallery: wide shots for counting, close-ups for labels. The pipeline is split on purpose: vision reads the lot, a comps step prices what was read, and plain deterministic arithmetic turns that into a number. The model never does the math. Fetching the full gallery moved make/model accuracy from 0.375 to 0.875 on a 24-lot hand-labelled bench. That is the single biggest lever I have found, bigger than any model swap. Most of what went wrong later was not misreading a badge. It was counting and itemizing: accessories sold as products, “X with Y” counted twice, a screen size parsed as a quantity. A final sanitizer can only tighten a verdict. And if the gallery fetch fails, the lot is deferred, not guessed. I wrote about how lots get found in the first place. This post is about the step after: given a lot that looks interesting, how do I know what is actually in the pile? ...

TL;DR I added an “Enhance” button to my photo viewer, backed by RealESRGAN running in ComfyUI on the Mac Studio. Its stated job was to help me read model plates and serial numbers in blurry surplus photos. I benched it against a plate where I knew the right answer. At small glyph sizes it did the opposite of its job: crisp, confident, wrong text. G1312B-60050 came back as 01D12B-(0)00. It is not a bad model pick. Four different upscalers garbled the same lines the same way. The usual advice (pre-upscale with a classical filter first) changed nothing. The failure is the generative prior, not a starved input. What I do now: the original pixels stay the source of truth, there is a text-safe sharpen view that invents nothing, and the AI view composites real pixels back over anything that looks like fine detail. A wrong model number off an enhanced photo is a wrong comp, which is a wrong bid. This one was worth catching before it spent money. Why I wanted it The most valuable pixels in a surplus lot photo are the smallest ones. The difference between a generic median price and an exact-model price is often a part number on a data plate, shot from a few feet away at a resolution where a 13-point label is about twelve pixels tall. I already pull the full gallery for every lot and archive it, so I had the best pixels the seller provided. They just were not enough for some plates. ...

TL;DR Before I merge anything where being wrong would be silent, I paste it into a live Gemini session and ask that model to attack it. Not “review this,” which gets you a rubber stamp with bullet points. Attack it. Over four days in April, that gate killed a performance optimization 88 minutes after it shipped, found two engine bugs that meant every historical number in my trading leaderboard was inflated, and identified five separate ways my “out-of-sample” evaluator was reading the future. Claude wrote all of that code. Claude also reviewed all of that code. Claude caught none of it. The useful property is not that Gemini is smarter, it is that Gemini did not write the thing and has no investment in the design being correct. ...

TL;DR I run an LLM-driven trading hypothesis engine. For a while, every result that came back looked too good — Sharpe ratios above 5, win rates above 70%, all on out-of-sample windows. They were lies. The model was reading dates, headlines, and tickers in the prompt and pattern-matching against its training data, which extends well past my “out-of-sample” cutoff. The fix was a masking layer I now call Blind Oracle: strip every leak before evaluation, run the trigger before the eval, gate promotion on out-of-sample Sharpe with the masking enforced. After it shipped, the inflated numbers collapsed back to honest reality. Some hypotheses survived; most didn’t. That’s exactly what I needed to know. ...