Reading a surplus lot from its photos: the identification pipeline

TL;DR

• Resale is the hobby that funds the hobby, and the expensive mistake in it is not a bad price. It is a confident wrong model number, because everything downstream (comps, margin, bid ceiling) is built on it.
• Auction search results give you one thumbnail and a title the seller typed in a hurry. The real information is in the gallery: wide shots for counting, close-ups for labels.
• The pipeline is split on purpose: vision reads the lot, a comps step prices what was read, and plain deterministic arithmetic turns that into a number. The model never does the math.
• Fetching the full gallery moved make/model accuracy from 0.375 to 0.875 on a 24-lot hand-labelled bench. That is the single biggest lever I have found, bigger than any model swap.
• Most of what went wrong later was not misreading a badge. It was counting and itemizing: accessories sold as products, “X with Y” counted twice, a screen size parsed as a quantity.
• A final sanitizer can only tighten a verdict. And if the gallery fetch fails, the lot is deferred, not guessed.

I wrote about how lots get found in the first place. This post is about the step after: given a lot that looks interesting, how do I know what is actually in the pile?

The problem with a lot title

Government and corporate surplus is sold in lots, and lot titles are written by someone clearing a loading dock. “Lab equipment, assorted.” “Lot of 10 computers.” “Dell with monitor.” The description lies by omission, and sometimes by outright error: a title will claim one model when the nameplate in photo seven says another.

The search listing makes it worse. The bulk search response carries one primary thumbnail per lot. If you evaluate from that, you are guessing at a pile from a single angle, which is what my first version did. It looked fine in the demo and was quietly wrong in production.

So the first rule of the pipeline: the title is a claim to verify, not a fact.

Stage 1: get every photo

For each lot worth a closer look, a second request hits the lot’s detail endpoint (no browser, plain HTTP with the session headers) and returns the full gallery, up to 40 photos. Then:

1. Fetch with browser-like headers. The image CDN returns 403 to a default client user agent.
2. Normalize everything to JPEG. More on why in the failure list, but the short version is that the CDN serves WebP even for URLs ending in .jpg, and one of my local runtimes refuses to decode WebP.
3. Archive the gallery to object storage. The CDN drops a lot’s images after the auction ages out, and I want to replay old lots against new code. If I do not keep the photos at evaluation time, I cannot reproduce an evaluation later. This step is best-effort and never blocks the evaluation.

The normalization is the least glamorous function in the project and it earned its place:

def _to_jpeg_b64(content: bytes) -> str | None:
    """Normalize fetched image bytes to JPEG base64 for the vision model."""
    try:
        im = Image.open(io.BytesIO(content))
        if im.mode != "RGB":
            im = im.convert("RGB")
        buf = io.BytesIO()
        im.save(buf, "JPEG", quality=85)
        return base64.standard_b64encode(buf.getvalue()).decode()
    except Exception:
        return None  # that photo is skipped, the rest still go through

A photo that cannot be decoded is dropped. The read proceeds on the others.

Stage 2: one read, many photos, plus the paperwork

The vision call gets all the photos at once, in one prompt, with the instruction that they are multiple angles of the same lot: combine wide shots (for unit count) with close-ups (for labels and badges), and prefer a readable label over a guess. The lot title and description go in as context, explicitly labelled as claims.

Many lots also carry manifest attachments: spreadsheets, PDFs, sometimes a Word document listing exact models and serials. Those are extracted to text (a handful of files, size-capped, row-capped) and put in the prompt flagged as authoritative. A model named in the manifest beats a model guessed from a blurry photo. Scanned PDFs get rendered to page images so the same vision read handles them.

The prompt asks for four things: identify one canonical make and model (never a comma-joined list, and null if unclear), itemize genuinely mixed lots, count distinct physical units, and assess condition. It also asks for size and shippability, because a lot I cannot fit in the car is not a lot I can buy.

A review pass, only when needed

If the primary read finds nothing at all (no make/model and no usable unit list), a second pass re-reads the same photos with a different, local model. Only then. I tried a full cascade where every read got reviewed, and it scored worse than the direct read, because a review framing anchors the reviewer on the first answer. So the review is a backstop for empty results, nothing more. In production it rarely fires.

Stage 3: identifiers in plain code

Serial numbers, date codes, calibration stickers, expiry dates and hazard labels are not a job for a model’s judgment. The vision read is asked to transcribe them into an identifiers field, and then ordinary Python decodes them: manufacturer serial formats, sticker dates, hazard markings. A radioactive or biohazard label decoded from the photos is a hard skip, no discussion.

The point of putting this in code is that it is testable. I can write a test case for a serial format and be certain what it does. I cannot do that for a prompt.

The output schema

The first version asked for a flat object: make_model, quantity, condition, and optional line_items for mixed lots. Version two kept those and added the fields that came from watching it fail:

{
  "make_model": "BRAND PRODUCT-LINE MODEL-NUMBER" or null,
  "quantity": 4,
  "condition": "Rating: Good\nIssues: ...",
  "units": [
    {"product": "...", "brand": "...", "model": "...",
     "qty": 4, "confidence": "...", "evidence_photo": 3}
  ],
  "accessories": ["..."],          // listed, never priced
  "excluded_adjacent": ["..."],    // the neighbouring pallet in the photo
  "identifiers": {
    "serials": [], "model_numbers": [], "date_codes": [],
    "cal_sticker": null, "asset_tags": [], "hazard_labels": [],
    "expiry_dates": []
  },
  "model_confidence": "read | inferred | unknown"
}

Two fields matter most.

model_confidence is the model’s own statement of how it got the identification. Read means it saw the model number on a label. Inferred means it deduced it from the shape, the badge family, the title. Unknown means it could not establish one. Inferred identifications get a discount downstream, and a model number the read did not actually see is not allowed to become a hard gate in the comps step. If the confidence is not read, the pipeline crops the label region out of the original photo, enlarges it, and re-reads it, bounded to a couple of extra calls per lot.

excluded_adjacent and accessories exist because of the failure list below. Giving the model a place to put the things it should not count is better than telling it not to count them.

quantity is the sum of the unit quantities, computed, not asserted.

The split that makes it safe

The architecture is three stages with a hard boundary between each:

photos + manifest  ->  vision read  ->  structured identification
identification     ->  comps lookup ->  priced comparables
comparables + condition + quantity  ->  deterministic arithmetic  ->  verdict

The model identifies. Comps are a data lookup. The money math is a plain function that I have verified to the cent by recomputing a lot by hand. When I audited a bad batch of valuations, every error entered upstream of the arithmetic: wrong model, wrong count, wrong comp set. None were in the multiplication.

Then there is a sanitizer at the single chokepoint every evaluation passes through before it is saved. It is deterministic, and it can only tighten: cap a bid, demote a BUY to WATCH, demote a verdict whose median rests on no real comps. It has no path to make a verdict more aggressive. That asymmetry is the whole design. A model that is wrong in the optimistic direction costs real money; a rule that is wrong in the cautious direction costs me a look.

And one more safety: if the gallery fetch fails, the lot is deferred. An expired session token once meant lots were being scored from the single thumbnail, and the board showed a confident verdict built on “no model number identified”. Now a failed fetch produces no evaluation at all, and the lot is retried. A missing verdict is honest. A confident verdict from one photo is not.

What one photo versus the whole gallery is worth

I benched this on 24 real lots with hand-labelled make and model, scored by a deterministic brand and model-number match, with a model judge only for borderline cases. Same model, same prompt with the title, two conditions:

Different models landed in a tight cluster once they had the gallery, within noise on 24 lots. The model choice mattered much less than whether it could see the label. Why 24 lots is a small bench, and how I measure changes anyway, is a follow-up post.

How it went wrong

This is the part I wish I had read first. Each of these was a real production failure.

Accessories itemized as products. A conference camera lot and a handheld console lot both came back with the stands, skins and charging cables as separate priced lines. The fix was a keyword guard on line items plus the accessories field that is never priced.

“X with Y” counted twice. A lot titled like “analyzer with centrifuge” is one system with a companion part. The pipeline itemized both, summed two medians, and, on an empty comps list, produced a four-figure recommendation. Only a hard bid ceiling stopped it. The title check now treats “with” joiners as one system, and it is deliberately conservative: it can only lower a valuation.

A screen size parsed as a quantity. A booth lot with “27 x 15.6 viewing area” in the title came out as a quantity of 15. Dimension chains are now blanked before quantity hints are considered, and only explicit counts like “lot of 10” can override what the photos show.

A lab instrument priced off trading-card comps. A liquid chromatography module was matched to comps by name words, and the median came out at about a dollar, from collectible card listings with a similar word in the title. Now the comp set has to fit the item: required brand, required model tokens, a head-noun check, and a “do these comps look like this thing” judgment that can return no match.

WebP breaking a local runtime. The CDN serves WebP regardless of the extension. The local runtime answered 400 Failed to load image and the lot got no read. Hence the JPEG normalization above.

Reasoning models returning empty content. A model with a thinking budget spent its whole token allowance reasoning and returned an empty string. The call now has a bigger budget, JSON mode, and an explicit guard for empty content, which is treated as a failed call and not as “nothing in the lot”.

Truncated output inventing brands. The default output limit on a local model cut off long ten-photo answers mid-JSON, and a truncated draft cascaded through the review step into invented brands. Raising the limit fixed it. The lesson was to treat truncation as a failure, not as a shorter answer.

A comps outage nobody noticed for five days. The sold-comps source died, and its response was valid-looking and empty. An empty response was interpreted as “there is no market for this”, so for five days a stream of lots were marked as having no comps. Now a response missing its expected structure counts as a vendor failure, never as zero results. An outage should never look like a market opinion.

The volume, and what it does and does not mean

When I last counted, in early October, the evaluations table held about 79,500 evaluation rows covering roughly 35,900 distinct lots. That is not 79,500 vision reads. Lots get re-evaluated as prices move or when I click re-evaluate, and many lots are filtered out by deterministic rules before any model is called (vehicles, heavy equipment, and so on).

I will not claim a reliability number from that volume. I have no stored comparison of predictions against realized sale prices, so I cannot say how often the identification is right in the wild. What I can say: make/model reading, with the gallery, is good. End-to-end valuation is much weaker, because the errors live in counting and comps. A hand audit of a batch of lab lots changed the verdict on most of them. Every one of those changes was upstream of the arithmetic.

What I would do the same, and what I would not

Same: keep the model out of the math, keep the sanitizer one-directional, archive the photos, treat the title as a claim.

Different: I would build the golden set before the pipeline, not after. Every failure above is now a test fixture, but each one cost a bad recommendation first.

An earlier post covered one fix I was proud of that turned out to make things worse: running super-resolution on model plates.