Kubernetes

TL;DR I joined eight surplus all-in-ones to my office k3s cluster as workers (the build is in the previous post). Six things went wrong, and all six failed quietly. A scheduling taint left 96 cores running zero workload pods while three small VMs ran 66. Prose inside a YAML folded scalar became k3s command-line arguments, and k3s silently dropped every flag after it. A DHCP lease and a pinned node IP disagreed. The login greeter suspended WiFi-only machines after 20 minutes. One mokutil command broke TPM auto-unlock. A weekly patch play skipped its own job for weeks. The common fix is boring: read back what the system actually did and assert on that, not on the exit code. 1. The taint that idled the fleet My original plan, from a proposal doc, was to taint the panels so nothing but the kiosk would run on them. What shipped was a softer version, PreferNoSchedule, because I also wanted Longhorn on the panels, and Longhorn’s DaemonSets would be locked out by a hard taint. Changing Longhorn’s toleration setting needs every volume detached, which means downtime just to satisfy a taint. ...

TL;DR I bought nine Dell OptiPlex all-in-ones (a mix of 7760 and 7780) in one government surplus lot for $1,209.37 all-in, about $134 a unit. Tax and shipping were zero; I picked them up myself. They run Debian 13 with a LUKS-encrypted root that unlocks through the TPM, Secure Boot on, and a signed NVIDIA module. Getting all three to coexist was most of the work. Eight of them joined my second, small k3s cluster (the “office cluster” at the shop where I do resale and bench work) as workers. Together they add 96 cores and 170 GiB of allocatable memory to a cluster whose three agent VMs had 12 cores and 23 GiB between them. The join is a dedicated Ansible playbook that reuses the existing k3s_agent role. One canary first, then the other seven in two waves. I didn’t measure power draw on these, so there’s no watts figure in this post. The older OptiPlex desktop cluster number doesn’t transfer. Why all-in-ones Most of what I buy for the hobby that funds the hobby is resale stock. This lot was different: nine machines, all wiped, “last powered on six months ago”, sold as-is, pickup by appointment at a school. The seller doesn’t box or ship. The hammer price was $1,075.00, plus a $134.37 buyer premium. I allocated the cost per unit in the inventory database and marked all nine as kept rather than for sale. ...

TL;DR Everything in my resale pipeline runs on CronJobs, which is correct until the moment you want fresh data now and the next run is 40 minutes out. Fix: a refresh button on each dashboard that creates a Kubernetes Job from the existing CronJob’s template, via the API, using a dedicated ServiceAccount with a tightly scoped Role. The web app gets permission to create Jobs in one namespace and nothing else. No cluster-admin, no shelling out to kubectl, no shared token. Six dashboards: opportunities, active bids, sold, listings, pickups, contacts. Every table sorts on every column, which is not a nice-to-have — unsorted tables hide your worst-performing categories. The metric that changed my buying wasn’t margin. It was days from sourced to sold, by category — and it says two categories I liked were quietly eating all my working capital. Cron is right until you’re standing in front of the shelf The pipeline is scheduled work. Crawlers every six hours, evaluation hourly, order sync every six hours, pack slips twice an hour. That’s the right architecture: it’s cheap, it’s resilient, it recovers on the next tick, and nothing depends on me being awake. ...

TL;DR The work between “I own this thing” and “money arrived” is mostly paperwork and printing, and it’s where the hours actually go. Listings get built through the marketplace API, not by driving a browser — structured item specifics as JSON beat fighting a typeahead widget every time. Drafts are created automatically. Publishing is always manual. Publish is a money-moving, buyer-facing, irreversible action; it gets a human every time. Barcode labels render from the inventory record itself: ZPL → PDF → print job to a thermal printer, triggered from the item’s page. No separate label app, no retyping serials. Pack slips print on a schedule, and the job verifies the printer’s page counter moved before marking the slip printed. Optimistic “it printed” flags are how orders ship without paperwork. Anything bulk or financial — repricing, printing five-plus labels — sits behind an explicit confirmation. Cheap friction in exactly the right places. The boring half is the expensive half Sourcing gets all the attention. I’ve written about the funnel that finds lots and the data model that tracks the units. Both were interesting problems. ...

TL;DR Government and institutional surplus auctions are a firehose: thousands of live lots, most of them badly photographed, vaguely titled, and worthless. Browsing them by hand doesn’t scale and — worse — it’s biased. You click the pretty photos, and the pretty photos are the ones everyone else clicked. The fix is a funnel: ~24 scheduled crawlers feeding a single Postgres table, photos and paperwork in object storage, an hourly evaluation pass, and one ranked board. The judgement layer (reading the goods out of photos, estimating what they’re worth) runs as a small model pipeline with a second pass that checks the first. I’m deliberately light on its internals here. The output isn’t a decision. It’s a shortlist with reasons attached — and a “this is wrong” button that puts the lot back in the queue. The single highest-value change in three months was raising how many photos get looked at. The label is often in photo nine. None of this is my job. It’s the hobby that pays for the hobby, which is the only reason it’s allowed to be this over-engineered. Why I built a data pipeline for a weekend habit Let’s be honest about the proportions here. I built two dozen scheduled crawlers, an object store, an hourly evaluation pass and a ranked dashboard, in order to buy second-hand equipment on the internet slightly better than I did before. ...

TL;DR My production homelab runs on Lenovo M920q tinies, and I still think those are the sweet spot. But if I were starting over today with a tight budget, I’d buy a stack of government-surplus Dell OptiPlex 7060 and 7070 desktops instead. They go for around $150 each refurbished — 6-core 8th/9th-gen Intel, an SSD, and Windows 11 already on them — and they make excellent Kubernetes nodes with exactly two cheap upgrades: a bit more RAM and a second network card. ...

TL;DR Every service I run — ComfyUI, Grafana, Vault, even the ancient app on a Mac across the network — lives behind a Traefik forward-auth middleware that hands off to Authentik. No per-service login page. One Authentik login shared across everything. The magic is a two-route IngressRoute pattern: a protected route with the middleware + an unprotected callback route for the OAuth flow itself. Adding a new service to the cluster takes five lines of YAML. Wiring a non-Kubernetes backend — like the Mac that runs ComfyUI and Ollama — takes a service-with-manual-endpoints proxy. ...

TL;DR A Mac Studio M3 Ultra costs the same as a single 4090 but comes with 256 GB of unified memory and 60-core GPU, all running at 100–200 W under inference. I stopped trying to pass MPS into containers and instead run Ollama and ComfyUI natively on macOS, then proxy them back into k3s as simple Kubernetes Services with manual Endpoints. Two Mac Studios connected via Thunderbolt 5 split the load: one handles hot-path LLM inference and embeddings, the other runs the heavy forge for diffusion and long-horizon reasoning. Both are cheaper to run than a single-socket A100 and require no special driver stacks. ...

TL;DR Three weeks after accidentally wiping GitLab with a misdirected blkdiscard and rebuilding from S3, I scheduled a deliberate drill: wipe GitLab, Vault, Harbor’s proxy cache, Authentik’s database, and one Longhorn volume on a Saturday morning, then rebuild everything from Terraform + S3 with a stopwatch running. Total drill time: 4 hours 22 minutes, end to end. About 90 minutes of that was actual rebuild work; the rest was discovering pieces of state I’d accidentally left out of the IaC. ...

TL;DR A year ago my stack was the usual mix — GitHub for code, ECR for images, GitHub Actions for CI, Docker Hub for upstreams, Route53 + S3 + CloudFront for the blog. Most of that’s still where it should be. About a third of it isn’t. This post is the retrospective on what came home, what stayed rented, and the rule of thumb I now use when deciding which side of the line a new service goes on. The short version: self-host the things you operate; rent the things you’d never have time to operate. ...