Benching a vision pipeline on 17 lots without fooling myself

TL;DR

• My pipeline reads photos of surplus lots, finds comparable sales, and does arithmetic to produce a bid ceiling. To change any part of it safely, I need a measurement that can tell a real improvement from luck.
• The method: a small golden set of lots with human-verified answers, paired comparisons (every candidate sees the same lots), an exact McNemar test on the cases where candidates disagree, and Wilson intervals on every rate.
• On 17 lots, “accuracy %” is noise. A single lot moves the number by about six points, and the 95 percent interval on a respectable-looking score is roughly thirty points wide.
• Reading make and model is the easy part. Counting units and choosing comps is where the error enters, so I score those separately from identification.
• The measurement that actually changed my mind was not a benchmark at all. It was re-running a batch of lab lots by hand: the summed bid ceilings dropped by about 72 percent.
• No per-model leaderboard here, on purpose. The method is the transferable part, and a small bench makes model rankings mostly an artifact.

This is the method behind the numbers in the identification post. Where that post shows how the pipeline works, this one is about how I know whether a change made it better.

The golden set

A golden set is a collection of cases with answers I trust more than the system’s. Mine comes in two kinds.

Identification cases. Real lots, labelled with the make and model that the item actually turned out to be. The strongest labels are lots I won and physically hold, because I can read the nameplate in person. A weaker tier is lots I hand-evaluated carefully from the photos. Keeping the tiers separate matters: a label from my own bench is ground truth, a label from my own squinting at the same photos the model sees is only a second opinion.

Valuation cases. Seventeen lots, stored as fixtures in the repo, with a human-verified value. The bench replays recorded model answers offline in CI, so a change to the comps logic or the arithmetic is tested on every merge request without a model call. A separate live mode re-runs the actual vision step when I want to test the whole thing.

Offline replay is the part I would copy first. It turns “did my change to the comps filter break anything” from a half-day experiment into a test that runs in seconds. It also means I can change the deterministic half of the pipeline without any model variance at all, because the model’s answers are frozen.

Scoring: hit, abstain, wrong

Identification is scored in three states, not two:

• Hit: the prediction shares a distinctive model token with the label. “Dell OptiPlex 7090 SFF” and “OptiPlex 7090” are the same machine, so exact string equality would be wrong.
• Abstain: the model returned nothing, or named a category without a model.
• Wrong: it named a model, and the model was not the label.

The split matters because abstaining and being wrong have different costs. An abstention gets a human look. A confident wrong answer becomes a comp search, a bid ceiling, and potentially money. A single accuracy number that folds those together hides the thing I care about most, so I report the wrong rate by itself.

Paired comparisons and McNemar

Suppose I want to know whether prompt B beats prompt A. The naive approach: run each on the lots, compare accuracy. That wastes information. Both runs see the same lots, so a lot that is easy for both tells me nothing about which is better. All the information is in the discordant pairs: lots where A hit and B missed, and lots where B hit and A missed.

McNemar’s test looks only at those. Under the null hypothesis that the two are equally good, each discordant lot is a coin flip, so the count follows a binomial with p = 0.5. For small samples I use the exact binomial two-sided p-value, not the chi-square approximation, which misbehaves when the discordant counts are small. The core of it is a few lines:

n_disc = a_only + b_only
lesser = min(a_only, b_only)
tail = sum(math.comb(n_disc, i) for i in range(lesser + 1)) / (2 ** n_disc)
p = min(1.0, 2 * tail)

To see why this discipline matters, consider an example. B beats A on three lots and loses on none. That feels decisive. The exact p-value is 0.25. Five wins to one loss is 0.22. You need six wins and no losses to get under 0.05. On a bench this small, most of the improvements I am excited about cannot be distinguished from noise, and the test says so.

The reverse failure also exists. If a change does nothing on 17 lots, that does not show it does nothing. It shows I could not have seen it.

Wilson intervals, and why 17 is noise

Every rate gets a Wilson 95 percent interval, which behaves at small n where the textbook normal approximation gives nonsense like intervals that exceed 100 percent. Some real numbers from my benches:

Look at the second row. Fifteen of 17 sounds like “about 90 percent.” The interval says anything from two thirds to nearly everything is compatible with that result. And the last row is the one that bites: two runs that land one lot apart on 17 have intervals that overlap across nearly their whole width. Any leaderboard built from a difference like that is a coin flip with a label on it.

The honest summary for a bench this size: it is good for regression detection and failure discovery, and nearly useless for ranking close candidates. I use it for the first two.

The in-band definition, for the valuation cases, is the value ratio against the human-verified answer landing between half and one and a half times. It is a coarse band on purpose. I care about “roughly right,” and a precise-looking error metric would suggest precision the comps cannot support.

Reading is fine, counting and comps are where it breaks

The most useful finding from all this was not about any candidate. It was about where the error lives.

On identification, once the model has the full gallery, hit rates sit in the high eighties and nineties on 24 hand-labelled lots. Different candidates cluster inside each other’s noise. This was the one photo versus full gallery result, where the jump from 0.375 to 0.875 came from giving the model more evidence, not from choosing a better model.

On end-to-end valuation, the in-band rate was far lower than the identification rate. My first live run on the 17 lots landed 4 in band, with a median value ratio of 0.56 and the worst lot off by 6x. After a round of fixes to the comps layer, replaying the same recorded answers offline moved the median ratio from 2.85 to 0.88, 15 of 17 in band, and the worst lot’s error from 31x to 2.6x.

Notice what that is: the model’s identifications were identical between the before and after. The improvement came entirely from deterministic code downstream, and the offline replay is the only reason I can say so with a straight face. The failures were things like spec words (an “8-channel” in a title) being used as model tokens, a brand never being required for a comp match, a hedged “e.g.” model number becoming a hard gate, and accessories counted as units. None of them are vision problems. When people say a vision pipeline “has an accuracy of X,” the number usually describes the easy half.

So I now score three things separately: did it read the right model, did it count the right number, and did the comps fit the item. A blended score hides which one regressed.

The audit that was not a benchmark

The most persuasive evidence I have came from a hand audit. I re-ran 125 lab-equipment lots with full-resolution photo review, more than fifteen hundred photos by eye, and compared my corrected evaluations against what the automated pipeline had produced.

• The automated verdict was changed on about 80 of the 125 lots.
• The count of lots showing as actionable (buy or watch) dropped from 88 to 58.
• The sum of the actionable bid ceilings fell by about 72 percent.
• Of 22 lots the pipeline had marked as a buy, only 4 stayed a buy.

The causes are the failure list from the identification post: a screen dimension read as a quantity, a model number in the title that differed from the one on the plate, one configured system counted as four instruments, an instrument priced off trading-card listings. Every one entered upstream of the arithmetic, which I had separately verified correct to the cent.

This audit is a different kind of tool than the golden set. It is expensive, it is not repeatable on every commit, and it gives you a population-level estimate of how wrong the pipeline is rather than a pass/fail. But it found things the golden set could not, because the golden set only contains the failures I had already thought of. After the audit, each root cause became a fixture or a test, which is how a one-off human review turns into a standing regression check.

I think the two are complementary: audits discover the failure classes, the golden set prevents their return.

What I would tell someone starting one

1. Label with the strongest evidence you have, and tier it. Items you physically hold beat photos you squinted at.
2. Pair everything. Same cases for every candidate, test only the discordant ones.
3. Put an interval on every rate. If you would not be surprised by a result a few lots either way, the number is not a measurement.
4. Score abstain separately from wrong. They are different costs.
5. Score each stage separately. An end-to-end number averages a stage that works with one that does not.
6. Freeze the model’s answers and replay them. Then the deterministic half of your pipeline is testable in CI with no variance.
7. Run an occasional expensive audit. The set only knows what you have already learned to be afraid of.
8. Be wary of rankings from a small set. Mine is a smoke detector. It is not a scale.

I started this because I trusted a number I should not have. I now keep the bench small, the intervals visible, and the claims modest, which is annoying and which is also the point.