Operations

TL;DR The dangerous remote failure isn’t the one that errors. It’s the one that reports success — exit code 0, green tick, desktop loads, nothing in the logs. Five real ones from the last two months, each with the wrong hypothesis I actually held. Your diagnostic tool can be the thing lying to you. A packet capture showed traffic arriving on a port that was being dropped, because capture taps before the firewall does. Provisioning needs sentinel files, not vibes. “It looks finished” is not a completion signal; a file the process writes at the end is. Where agents genuinely help: assembling the four facts a diagnosis needs, and recalling the failure signature from three weeks ago. Where they hurt: confidently narrating a plausible cause for a symptom they can’t see. Every one of these produced a page in the knowledge base with the wrong hypothesis recorded, because that’s the field that saves the next hour. Success is not a signal I run machines I can’t put my hands on: a bench of surplus desktops at an office across town, cluster nodes, a couple of workstations belonging to other people. Remote hands come from a KVM device and SSH, and I’ve written about that hardware setup already. ...

TL;DR Everything in my resale pipeline runs on CronJobs, which is correct until the moment you want fresh data now and the next run is 40 minutes out. Fix: a refresh button on each dashboard that creates a Kubernetes Job from the existing CronJob’s template, via the API, using a dedicated ServiceAccount with a tightly scoped Role. The web app gets permission to create Jobs in one namespace and nothing else. No cluster-admin, no shelling out to kubectl, no shared token. Six dashboards: opportunities, active bids, sold, listings, pickups, contacts. Every table sorts on every column, which is not a nice-to-have — unsorted tables hide your worst-performing categories. The metric that changed my buying wasn’t margin. It was days from sourced to sold, by category — and it says two categories I liked were quietly eating all my working capital. Cron is right until you’re standing in front of the shelf The pipeline is scheduled work. Crawlers every six hours, evaluation hourly, order sync every six hours, pack slips twice an hour. That’s the right architecture: it’s cheap, it’s resilient, it recovers on the next tick, and nothing depends on me being awake. ...

TL;DR An unmodelled repair backlog becomes a pile of dusty things you feel bad about. Ask me how I know. Four ticket states: open, blocked, done, abandoned. blocked and abandoned are the two that do the work. abandoned is a feature. A queue you can’t quit is a queue that grows forever, and “I’ll get to it” is not a state. Parts are modelled as purchases and uses separately, because you buy a bag of ten capacitors and use two. Otherwise your cost-per-repair is nonsense. Labour gets logged per worker with an hourly rate, and that number flows all the way back into what I’m willing to bid on a lot. The no-go rule does more for profitability than any repair skill I have: if parts and time exceed roughly 60% of realistic resale, or resale is under about $120, it doesn’t get a ticket. Other people work this queue. Four access roles keep them on inventory and tickets and away from anything financial. Why a pile forms Buying surplus means buying broken things, on purpose, because broken is cheap and some broken things are trivially fixable. A cassette deck with a perished belt is a $4 part and twenty minutes. A workstation that won’t POST is often one reseated stick of RAM. ...

TL;DR The work between “I own this thing” and “money arrived” is mostly paperwork and printing, and it’s where the hours actually go. Listings get built through the marketplace API, not by driving a browser — structured item specifics as JSON beat fighting a typeahead widget every time. Drafts are created automatically. Publishing is always manual. Publish is a money-moving, buyer-facing, irreversible action; it gets a human every time. Barcode labels render from the inventory record itself: ZPL → PDF → print job to a thermal printer, triggered from the item’s page. No separate label app, no retyping serials. Pack slips print on a schedule, and the job verifies the printer’s page counter moved before marking the slip printed. Optimistic “it printed” flags are how orders ship without paperwork. Anything bulk or financial — repricing, printing five-plus labels — sits behind an explicit confirmation. Cheap friction in exactly the right places. The boring half is the expensive half Sourcing gets all the attention. I’ve written about the funnel that finds lots and the data model that tracks the units. Both were interesting problems. ...

TL;DR Giving an agent write access to a wiki is a twenty-minute job. The interesting work is stopping it from filling the wiki with plausible sludge. Four page types, each with a contract about what it must contain. An agent writing a runbook and an agent writing a reference page are doing different jobs and need different rules. Staleness is a field, not a vibe. Every page carries who wrote it, when, and how it was verified. Unverified pages are visibly second-class. The write gate is an environment variable, off by default. Read is free; write is a decision. Incidents become pages, or the incident was wasted. That’s the one rule I’d keep if I could only keep one. Things a wiki can’t hold — PDFs, images, rendered one-pagers I need to share with someone outside the cluster — go to a small auth-gated artifact host instead. Don’t paste binaries into a wiki. The easy part I have a self-hosted wiki with a GraphQL API and a few hundred pages. Wiring an agent into it took about two hundred lines of Python — a small MCP server with search, read, list, create, and update, flattening the API’s nested responses into markdown so the agent gets text instead of JSON soup. I wrote about the mechanics of that server, and two others, in the MCP server cookbook. ...

TL;DR You buy a lot. You sell units. That asymmetry is the whole data-modelling problem, and spreadsheets lose to it immediately. The spine is four tables: purchase_lot → inventory_item → listing → sale, with join tables where the relationships are genuinely many-to-many. Cost gets allocated from the lot down to units, and the allocation locks once the first unit sells. Retroactively re-allocating cost after a sale is how you accidentally rewrite last quarter’s profit. A unit walks a status machine: sourced → processing → drafted → listed → sold → shipped → reconciled, with dead and personal as the two honest escape hatches. Selling quantity-N on one listing is N sales, not one sale with a quantity column. I modelled it the wrong way first and spent August unpicking it. Packing several units into one sellable bundle, and splitting a bundle back into units, both need to conserve cost exactly. Getting that wrong is silent and it shows up as fake margin. The mismatch Here’s the transaction that starts everything: I pay $340 for a pallet described as Assorted Networking and Computer Equipment. One line item, one payment, one date. ...

TL;DR I maintain workstations for people who are never going to run git pull — so the config lives in a repo I own and they never see. It is a reference/restore repo, not config management: per-machine folders hold snapshots of what’s actually installed, and restore is a deliberate install -Dm755, not a sync. The no-auto-sync part is the whole point. Silent convergence across machines that are supposed to differ is how you break someone else’s laptop from your desk. The boring 40 minutes — partitioning, encryption, sudo, base packages — is an unattended installer (a Debian preseed, now on its eleventh revision). The interesting 20 minutes is the per-machine folder. The agent’s job isn’t to apply config. It’s to read the repo, diff it against the live box, and tell me what drifted — then write the restore commands for me to approve. What I’d do differently: put the agent’s own profile in git before you have three machines copying it from each other over SSH. The problem with managing other people’s computers I run a handful of machines that aren’t mine. A couple of family desktops, a bench box at the office, and a laptop belonging to someone who does knowledge work and would rather not learn what a dotfile is. ...

TL;DR Government and institutional surplus auctions are a firehose: thousands of live lots, most of them badly photographed, vaguely titled, and worthless. Browsing them by hand doesn’t scale and — worse — it’s biased. You click the pretty photos, and the pretty photos are the ones everyone else clicked. The fix is a funnel: ~24 scheduled crawlers feeding a single Postgres table, photos and paperwork in object storage, an hourly evaluation pass, and one ranked board. The judgement layer (reading the goods out of photos, estimating what they’re worth) runs as a small model pipeline with a second pass that checks the first. I’m deliberately light on its internals here. The output isn’t a decision. It’s a shortlist with reasons attached — and a “this is wrong” button that puts the lot back in the queue. The single highest-value change in three months was raising how many photos get looked at. The label is often in photo nine. None of this is my job. It’s the hobby that pays for the hobby, which is the only reason it’s allowed to be this over-engineered. Why I built a data pipeline for a weekend habit Let’s be honest about the proportions here. I built two dozen scheduled crawlers, an object store, an hourly evaluation pass and a ranked dashboard, in order to buy second-hand equipment on the internet slightly better than I did before. ...

TL;DR Five rungs: Rung 0 (read-only) → Rung 1 (draft MRs) → Rung 2 (commit on /llm fix) → Rung 3 (auto-merge patches) → Rung 4 (exceptions-only). Promotion is earned, not promised — ~30 days at Rung 1 + <10% reverts → Rung 2; ~60 days at Rung 2 + <5% reverts → Rung 3. Infra agents cap at Rung 2 — Kubernetes, Terraform, secrets, cert-manager always need human approval to merge; blast radius is too high. Demotion is immediate on test deletion, secret leak, scope creep, or any deploy needing manual rollback. Non-negotiable gates (deterministic, run at the CI runner) block everything: no deleted tests, no secrets, no medium+ CVEs, destructive commands on allowlist only. From concept to operational I wrote about the trust ladder for agents in May — the philosophy of supervised → monitored → trusted → autonomous, and why you can’t just flip a switch. That post is the why; this one is the how. I’m going to walk through the exact rungs, the promotion criteria that actually work, the demotion triggers, and the CI rules that make it safe. ...

TL;DR Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not. Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%. Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex. Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs. Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive. The Problem: Autonomy Without a Leash Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill. ...