Homelab

TL;DR In July I wrote about a test-deletion sentinel: CI hard-fails if a merge request deletes tests, unless the commit message carries a [test-refactor] tag and an issue number. I was pleased with it. It turns out a tag in a commit message is a bypass that an agent (or I, on a bad day) can write in one line, and it excused every test change in the MR, not just the one I meant. ...

TL;DR I joined eight surplus all-in-ones to my office k3s cluster as workers (the build is in the previous post). Six things went wrong, and all six failed quietly. A scheduling taint left 96 cores running zero workload pods while three small VMs ran 66. Prose inside a YAML folded scalar became k3s command-line arguments, and k3s silently dropped every flag after it. A DHCP lease and a pinned node IP disagreed. The login greeter suspended WiFi-only machines after 20 minutes. One mokutil command broke TPM auto-unlock. A weekly patch play skipped its own job for weeks. The common fix is boring: read back what the system actually did and assert on that, not on the exit code. 1. The taint that idled the fleet My original plan, from a proposal doc, was to taint the panels so nothing but the kiosk would run on them. What shipped was a softer version, PreferNoSchedule, because I also wanted Longhorn on the panels, and Longhorn’s DaemonSets would be locked out by a hard taint. Changing Longhorn’s toleration setting needs every volume detached, which means downtime just to satisfy a taint. ...

TL;DR I bought nine Dell OptiPlex all-in-ones (a mix of 7760 and 7780) in one government surplus lot for $1,209.37 all-in, about $134 a unit. Tax and shipping were zero; I picked them up myself. They run Debian 13 with a LUKS-encrypted root that unlocks through the TPM, Secure Boot on, and a signed NVIDIA module. Getting all three to coexist was most of the work. Eight of them joined my second, small k3s cluster (the “office cluster” at the shop where I do resale and bench work) as workers. Together they add 96 cores and 170 GiB of allocatable memory to a cluster whose three agent VMs had 12 cores and 23 GiB between them. The join is a dedicated Ansible playbook that reuses the existing k3s_agent role. One canary first, then the other seven in two waves. I didn’t measure power draw on these, so there’s no watts figure in this post. The older OptiPlex desktop cluster number doesn’t transfer. Why all-in-ones Most of what I buy for the hobby that funds the hobby is resale stock. This lot was different: nine machines, all wiped, “last powered on six months ago”, sold as-is, pickup by appointment at a school. The seller doesn’t box or ship. The hammer price was $1,075.00, plus a $134.37 buyer premium. I allocated the cost per unit in the inventory database and marked all nine as kept rather than for sale. ...

TL;DR The screens on eight surplus all-in-ones show an ops board: bids, ship queue, pickups, repair backlog, cluster health, weather and traffic. They sit on desks around the office and do double duty: a workstation when someone sits down, the board the rest of the time. The same machines are on their way to becoming k3s worker nodes (one has joined so far), and that build gets its own post. It’s one static HTML page in Chrome kiosk mode. No framework server, no Kubernetes workload. Ansible deploys it. 15 scenes sit in a weighted registry and rotate off the wall clock, about 34 seconds each, so eight screens agree without talking to each other. When a fetch fails the page keeps the last good data and shows its age. A board with stale data and a warning beats a blank screen. It’s a different animal from my older home signage system, which is a k8s app with Raspberry Pi kiosks. One real thing it did: on 2026-09-06 the home cluster’s daily Longhorn backup had stopped completing, and the board’s home-cluster card showed the critical alert. Not the same project as the home signage I’ve written about digital signage on k3s before: an Angular app, a handful of Flask services, MQTT, Raspberry Pis driving displays. That’s a proper application and it lives in the home cluster. ...

TL;DR The dangerous remote failure isn’t the one that errors. It’s the one that reports success — exit code 0, green tick, desktop loads, nothing in the logs. Five real ones from the last two months, each with the wrong hypothesis I actually held. Your diagnostic tool can be the thing lying to you. A packet capture showed traffic arriving on a port that was being dropped, because capture taps before the firewall does. Provisioning needs sentinel files, not vibes. “It looks finished” is not a completion signal; a file the process writes at the end is. Where agents genuinely help: assembling the four facts a diagnosis needs, and recalling the failure signature from three weeks ago. Where they hurt: confidently narrating a plausible cause for a symptom they can’t see. Every one of these produced a page in the knowledge base with the wrong hypothesis recorded, because that’s the field that saves the next hour. Success is not a signal I run machines I can’t put my hands on: a bench of surplus desktops at an office across town, cluster nodes, a couple of workstations belonging to other people. Remote hands come from a KVM device and SSH, and I’ve written about that hardware setup already. ...

TL;DR Everything in my resale pipeline runs on CronJobs, which is correct until the moment you want fresh data now and the next run is 40 minutes out. Fix: a refresh button on each dashboard that creates a Kubernetes Job from the existing CronJob’s template, via the API, using a dedicated ServiceAccount with a tightly scoped Role. The web app gets permission to create Jobs in one namespace and nothing else. No cluster-admin, no shelling out to kubectl, no shared token. Six dashboards: opportunities, active bids, sold, listings, pickups, contacts. Every table sorts on every column, which is not a nice-to-have — unsorted tables hide your worst-performing categories. The metric that changed my buying wasn’t margin. It was days from sourced to sold, by category — and it says two categories I liked were quietly eating all my working capital. Cron is right until you’re standing in front of the shelf The pipeline is scheduled work. Crawlers every six hours, evaluation hourly, order sync every six hours, pack slips twice an hour. That’s the right architecture: it’s cheap, it’s resilient, it recovers on the next tick, and nothing depends on me being awake. ...

TL;DR An unmodelled repair backlog becomes a pile of dusty things you feel bad about. Ask me how I know. Four ticket states: open, blocked, done, abandoned. blocked and abandoned are the two that do the work. abandoned is a feature. A queue you can’t quit is a queue that grows forever, and “I’ll get to it” is not a state. Parts are modelled as purchases and uses separately, because you buy a bag of ten capacitors and use two. Otherwise your cost-per-repair is nonsense. Labour gets logged per worker with an hourly rate, and that number flows all the way back into what I’m willing to bid on a lot. The no-go rule does more for profitability than any repair skill I have: if parts and time exceed roughly 60% of realistic resale, or resale is under about $120, it doesn’t get a ticket. Other people work this queue. Four access roles keep them on inventory and tickets and away from anything financial. Why a pile forms Buying surplus means buying broken things, on purpose, because broken is cheap and some broken things are trivially fixable. A cassette deck with a perished belt is a $4 part and twenty minutes. A workstation that won’t POST is often one reseated stick of RAM. ...

TL;DR The work between “I own this thing” and “money arrived” is mostly paperwork and printing, and it’s where the hours actually go. Listings get built through the marketplace API, not by driving a browser — structured item specifics as JSON beat fighting a typeahead widget every time. Drafts are created automatically. Publishing is always manual. Publish is a money-moving, buyer-facing, irreversible action; it gets a human every time. Barcode labels render from the inventory record itself: ZPL → PDF → print job to a thermal printer, triggered from the item’s page. No separate label app, no retyping serials. Pack slips print on a schedule, and the job verifies the printer’s page counter moved before marking the slip printed. Optimistic “it printed” flags are how orders ship without paperwork. Anything bulk or financial — repricing, printing five-plus labels — sits behind an explicit confirmation. Cheap friction in exactly the right places. The boring half is the expensive half Sourcing gets all the attention. I’ve written about the funnel that finds lots and the data model that tracks the units. Both were interesting problems. ...

TL;DR Giving an agent write access to a wiki is a twenty-minute job. The interesting work is stopping it from filling the wiki with plausible sludge. Four page types, each with a contract about what it must contain. An agent writing a runbook and an agent writing a reference page are doing different jobs and need different rules. Staleness is a field, not a vibe. Every page carries who wrote it, when, and how it was verified. Unverified pages are visibly second-class. The write gate is an environment variable, off by default. Read is free; write is a decision. Incidents become pages, or the incident was wasted. That’s the one rule I’d keep if I could only keep one. Things a wiki can’t hold — PDFs, images, rendered one-pagers I need to share with someone outside the cluster — go to a small auth-gated artifact host instead. Don’t paste binaries into a wiki. The easy part I have a self-hosted wiki with a GraphQL API and a few hundred pages. Wiring an agent into it took about two hundred lines of Python — a small MCP server with search, read, list, create, and update, flattening the API’s nested responses into markdown so the agent gets text instead of JSON soup. I wrote about the mechanics of that server, and two others, in the MCP server cookbook. ...

TL;DR You buy a lot. You sell units. That asymmetry is the whole data-modelling problem, and spreadsheets lose to it immediately. The spine is four tables: purchase_lot → inventory_item → listing → sale, with join tables where the relationships are genuinely many-to-many. Cost gets allocated from the lot down to units, and the allocation locks once the first unit sells. Retroactively re-allocating cost after a sale is how you accidentally rewrite last quarter’s profit. A unit walks a status machine: sourced → processing → drafted → listed → sold → shipped → reconciled, with dead and personal as the two honest escape hatches. Selling quantity-N on one listing is N sales, not one sale with a quantity column. I modelled it the wrong way first and spent August unpicking it. Packing several units into one sellable bundle, and splitting a bundle back into units, both need to conserve cost exactly. Getting that wrong is silent and it shows up as fake margin. The mismatch Here’s the transaction that starts everything: I pay $340 for a pallet described as Assorted Networking and Computer Equipment. One line item, one payment, one date. ...