Benching a vision pipeline on 17 lots without fooling myself
TL;DR My pipeline reads photos of surplus lots, finds comparable sales, and does arithmetic to produce a bid ceiling. To change any part of it safely, I need a measurement that can tell a real improvement from luck. The method: a small golden set of lots with human-verified answers, paired comparisons (every candidate sees the same lots), an exact McNemar test on the cases where candidates disagree, and Wilson intervals on every rate. On 17 lots, “accuracy %” is noise. A single lot moves the number by about six points, and the 95 percent interval on a respectable-looking score is roughly thirty points wide. Reading make and model is the easy part. Counting units and choosing comps is where the error enters, so I score those separately from identification. The measurement that actually changed my mind was not a benchmark at all. It was re-running a batch of lab lots by hand: the summed bid ceilings dropped by about 72 percent. No per-model leaderboard here, on purpose. The method is the transferable part, and a small bench makes model rankings mostly an artifact. This is the method behind the numbers in the identification post. Where that post shows how the pipeline works, this one is about how I know whether a change made it better. ...