TL;DR

  • Government and institutional surplus auctions are a firehose: thousands of live lots, most of them badly photographed, vaguely titled, and worthless.
  • Browsing them by hand doesn’t scale and — worse — it’s biased. You click the pretty photos, and the pretty photos are the ones everyone else clicked.
  • The fix is a funnel: ~24 scheduled crawlers feeding a single Postgres table, photos and paperwork in object storage, an hourly evaluation pass, and one ranked board.
  • The judgement layer (reading the goods out of photos, estimating what they’re worth) runs as a small model pipeline with a second pass that checks the first. I’m deliberately light on its internals here.
  • The output isn’t a decision. It’s a shortlist with reasons attached — and a “this is wrong” button that puts the lot back in the queue.
  • The single highest-value change in three months was raising how many photos get looked at. The label is often in photo nine.
  • None of this is my job. It’s the hobby that pays for the hobby, which is the only reason it’s allowed to be this over-engineered.

Why I built a data pipeline for a weekend habit

Let’s be honest about the proportions here. I built two dozen scheduled crawlers, an object store, an hourly evaluation pass and a ranked dashboard, in order to buy second-hand equipment on the internet slightly better than I did before.

That is a ridiculous amount of infrastructure for a hobby. I’m not going to pretend otherwise. But the ridiculousness is the point, because a hobby is the correct place to build something like this:

  • The stakes are calibrated. If the pipeline is wrong, I overpay for a pallet of junk and I’m annoyed for a week. Nobody’s payroll depends on it. That’s exactly the risk level at which I’m willing to let an automated system make suggestions.
  • The feedback is free and it’s honest. The auction closes, somebody pays a number, and a few weeks later I find out whether my estimate was nonsense. No annotation project, no benchmark I had to invent. I get graded whether I like it or not.
  • The inputs are genuinely awful, which is the part you can’t simulate. Warehouse photos at bad angles, titles like Misc Electronics, scanned manifests. Every tidy tutorial dataset lies about this.
  • It funds itself. The cluster this runs on, the bench, the label printer, the parts drawer: all of it got paid for by things this pipeline found. That’s a nice loop and it’s the whole justification.

So yes, it’s overkill. It’s also the most instructive thing I’ve built, precisely because being wrong costs actual money rather than a red mark in a notebook.

Surplus is a volume problem disguised as a taste problem

I buy surplus. Government fleet departments, school districts, universities, hospitals, corporate liquidations. Somebody’s asset tag expires, the item goes on an auction site, and a week later it sells for a fraction of what it’s worth to someone who knows what it is.

The catch is that “knows what it is” is the whole game, and the auction sites are actively hostile to it. A lot is titled Misc Electronics. There are eleven photos, taken in a warehouse, at an angle, in fluorescent light. Somewhere in photo nine there’s a service label with a model number that determines whether this pallet is worth $60 or $2,600.

Close-up of a black IBM Wheelwriter 6 badge on a beige plastic machine housing, the model number legible against the textured panel

That’s the entire ballgame in one photo. Not the wide shot of the machine, not the title, not the description: the badge. “Electronic typewriter” is a $20 item. A specific electronic typewriter with a known-good keyboard assembly and a still-manufactured ribbon is not. Everything my pipeline does downstream of identification is arithmetic, and none of it means anything if this part is wrong.

For about a year I did this by hand. Coffee, three browser tabs, filter by category, scroll. It worked in the sense that I made money, and it failed in three ways I only saw once I had data:

  1. I looked at maybe 2% of the live lots in my radius. The other 98% might as well not have existed.
  2. I clicked the good photos. So did everyone else. Competition concentrates exactly where presentation is best, which is not where value is best.
  3. I had no memory. I’d evaluate a lot, lose the tab, and re-evaluate the same lot four days later from scratch, sometimes reaching a different conclusion.

None of those are taste problems. They’re pipeline problems.

The funnel

Four stages, all of it running in the cluster as CronJobs, all of it landing in one Postgres database.

discovery  →  storage  →  evaluation  →  shortlist
(~24 jobs)    (pg +       (hourly,       (one board,
              object)     batched)       ranked)

Discovery

Roughly two dozen scheduled crawl jobs, each scoped to one marketplace and one slice of it: general electronics in a geographic radius, lab and scientific equipment nationally, a municipal-auction site, two-way radios and comms gear, and so on. They run every six hours on staggered schedules.

Scoping them narrowly, rather than running one big crawler with a huge keyword list, turned out to matter. A crawler that covers one search zone has an obvious failure mode — it returns zero rows and you notice. A single mega-crawler that silently loses a category looks exactly like a slow week.

Each job writes lots into a shared table with the source, the source’s own lot reference, the closing time, the current bid, and the search zone that found it. The source lot reference is the dedupe key, and it earns its keep: the same physical pallet frequently appears under two search zones.

Storage

Photos and attached paperwork (manifests, condition reports, the occasional scanned inventory sheet) get pulled down at discovery time and stored in object storage — MinIO, in-cluster — rather than hotlinked.

That’s not just politeness. Auction photos disappear when the lot closes, and a closed lot with its photos still on hand is training data: it’s how I check, three weeks later, whether my estimate on a lot I lost was any good. Hotlinking would mean my entire evaluation history quietly rots.

One wrinkle worth knowing: the CDN serving those photos hands back WebP, and parts of my own pipeline wouldn’t take WebP. Everything gets transcoded to JPEG on ingest. Cheap, boring, and it eliminated a class of “why did this lot return nothing” bug that took me an embarrassing amount of time to find.

Evaluation

An hourly job picks up a batch of un-evaluated lots — ten per run — and produces a structured record for each: what the goods appear to be, how confident that read is, what it’s likely worth, what it’d cost to prep and ship, and a verdict.

I’m going to stay fairly high-level about the inside of this, because the specifics are the part I’d rather not hand over wholesale. The shape of it, though, is not a secret and is the part worth copying:

  • A vision pass reads the lot — photos plus any attached paperwork — and emits structured fields rather than prose. Make and model, quantity, visible condition, what’s missing.
  • A second pass checks the first one, and only fires when the first pass came back without a confident identification. It’s cheap insurance: it rescues a large majority of the cases the primary pass whiffed on, and it costs almost nothing because it doesn’t run on the lots that already worked.
  • Comparable sales come from an authorized data vendor, not from scraping. Realized prices, not asking prices. When the vendor has nothing, there’s a fallback path with a deliberate haircut applied, and the record notes which source it used.
  • The money math is deterministic. Once you have an identification and a comp, turning that into “what’s the most I should bid” is arithmetic — fees, shipping by weight band, prep labour, a buyer’s premium that varies per lot, and a margin buffer. No model involved. This is important: I want the judgement in the identification step, where it belongs, and nowhere near the arithmetic.

Every record keeps its provenance: which comp source, which pass produced the identification, whether a reviewer stage touched it. When a number looks wrong, I can see why it’s wrong rather than shrugging at a black box.

Shortlist

One board. Live lots only, evaluated, ranked by how much headroom there is between the current bid and what I’d be willing to pay. Filters for verdict, search zone, and closing-soon.

The detail view shows the reasoning, the photos, the comp source, and the arithmetic. Two buttons matter: flag (this evaluation is wrong, here’s why) and re-evaluate (try again, the photos updated or the first read was garbage). Both write to their own tables, so the flags accumulate into a record of where the pipeline is weak instead of evaporating into my memory.

What actually moved the needle

Three months in, ranked by impact:

1. Look at more photos. The evaluation pass originally read the first six photos. A quality sweep on lots I’d held in my hands found the identifying label in photos nine through eleven often enough to matter. Raising the window was a one-line config change and it was worth more than every prompt tweak combined.

The general lesson: when a vision pipeline is missing things, check what you’re not showing it before you start rewriting instructions.

2. Build a golden set from stuff you physically own. I have around seventy lots I actually bought, unpacked, and hand-labelled with what was really in them, plus a larger second tier of carefully reviewed evaluations. That’s the bench. Without it, “did that change help?” is a vibe. With it, it’s a number, and about a third of my clever ideas turned out to make things worse.

3. A cheap reviewer beats an expensive primary. I benched a few larger, pricier models as the main pass. The thing that actually cut the miss rate was keeping a modest primary and adding a second look that only fires on the cases the primary fumbled. Miss rate came down by more than half; cost barely moved.

What this doesn’t do

It doesn’t bid on its own. The board produces a shortlist and a ceiling; the decision to enter an auction is mine, every time. That’s not squeamishness about automation — it’s that a bid is an irreversible financial commitment, and the autonomy ladder says irreversible-and-financial stays manual until the track record is overwhelming. It isn’t yet.

It also doesn’t replace knowing things. The pipeline is very good at finding the pallet of lab gear and roughly bracketing its value. It has no idea that a particular instrument family always ships with a controller that’s missing from these photos, or that one manufacturer’s calibration certificate doubles resale. That’s still me, and the flag button is how I get it back into the system.

Gotchas

Defaults hide in your cost model. Buyer’s premium varies by source and by individual lot — anywhere from around 5% to nearly 20%. Mine is stored per lot with a fallback default, and for a long time only a minority of lots had a real value stored, meaning most were silently using the default. Every one of those estimates was wrong in the same direction. Audit your fallbacks; a default that’s used 57% of the time isn’t a default, it’s your model.

Asking prices lie, and they lie consistently upward. Active listings are what sellers hope for. Realized sales are what buyers paid. If you have to fall back to active listings, apply a haircut and record that you did, so the number carries a warning label.

A crawler that returns zero rows looks healthy. No errors, no exceptions, exit code 0. Alert on volume, not just failure — “this search zone found nothing for 24 hours” is the signal that matters, and I learned that after a site changed its markup and one zone quietly went dark.

Re-evaluation needs a queue, not a button that runs inline. Mine writes a request row that the next scheduled pass picks up first. Otherwise you’re holding a web request open for several minutes while a vision model thinks, and that fails in every boring way you’d expect.

Photos outlive listings, but only if you copied them. Say it twice because it’s a one-way door: once the lot closes, your chance to store the evidence is gone.

Where this sits now

Two dozen crawlers, an hourly evaluation pass, a single ranked board, and about a two-thirds reduction in time spent looking at auctions I was never going to bid on. The wins aren’t the lots it finds — it’s the lots it dismisses, at volume, so I only look at the ones with headroom.

Next up in this series: what happens after the hammer falls, when a pallet of assorted gear needs to become individual items with individual cost bases. That turned out to be the harder data problem.