Ansible

TL;DR I joined eight surplus all-in-ones to my office k3s cluster as workers (the build is in the previous post). Six things went wrong, and all six failed quietly. A scheduling taint left 96 cores running zero workload pods while three small VMs ran 66. Prose inside a YAML folded scalar became k3s command-line arguments, and k3s silently dropped every flag after it. A DHCP lease and a pinned node IP disagreed. The login greeter suspended WiFi-only machines after 20 minutes. One mokutil command broke TPM auto-unlock. A weekly patch play skipped its own job for weeks. The common fix is boring: read back what the system actually did and assert on that, not on the exit code. 1. The taint that idled the fleet My original plan, from a proposal doc, was to taint the panels so nothing but the kiosk would run on them. What shipped was a softer version, PreferNoSchedule, because I also wanted Longhorn on the panels, and Longhorn’s DaemonSets would be locked out by a hard taint. Changing Longhorn’s toleration setting needs every volume detached, which means downtime just to satisfy a taint. ...

TL;DR I bought nine Dell OptiPlex all-in-ones (a mix of 7760 and 7780) in one government surplus lot for $1,209.37 all-in, about $134 a unit. Tax and shipping were zero; I picked them up myself. They run Debian 13 with a LUKS-encrypted root that unlocks through the TPM, Secure Boot on, and a signed NVIDIA module. Getting all three to coexist was most of the work. Eight of them joined my second, small k3s cluster (the “office cluster” at the shop where I do resale and bench work) as workers. Together they add 96 cores and 170 GiB of allocatable memory to a cluster whose three agent VMs had 12 cores and 23 GiB between them. The join is a dedicated Ansible playbook that reuses the existing k3s_agent role. One canary first, then the other seven in two waves. I didn’t measure power draw on these, so there’s no watts figure in this post. The older OptiPlex desktop cluster number doesn’t transfer. Why all-in-ones Most of what I buy for the hobby that funds the hobby is resale stock. This lot was different: nine machines, all wiped, “last powered on six months ago”, sold as-is, pickup by appointment at a school. The seller doesn’t box or ship. The hammer price was $1,075.00, plus a $134.37 buyer premium. I allocated the cost per unit in the inventory database and marked all nine as kept rather than for sale. ...

TL;DR The screens on eight surplus all-in-ones show an ops board: bids, ship queue, pickups, repair backlog, cluster health, weather and traffic. They sit on desks around the office and do double duty: a workstation when someone sits down, the board the rest of the time. The same machines are on their way to becoming k3s worker nodes (one has joined so far), and that build gets its own post. It’s one static HTML page in Chrome kiosk mode. No framework server, no Kubernetes workload. Ansible deploys it. 15 scenes sit in a weighted registry and rotate off the wall clock, about 34 seconds each, so eight screens agree without talking to each other. When a fetch fails the page keeps the last good data and shows its age. A board with stale data and a warning beats a blank screen. It’s a different animal from my older home signage system, which is a k8s app with Raspberry Pi kiosks. One real thing it did: on 2026-09-06 the home cluster’s daily Longhorn backup had stopped completing, and the board’s home-cluster card showed the critical alert. Not the same project as the home signage I’ve written about digital signage on k3s before: an Angular app, a handful of Flask services, MQTT, Raspberry Pis driving displays. That’s a proper application and it lives in the home cluster. ...

TL;DR Three weeks after accidentally wiping GitLab with a misdirected blkdiscard and rebuilding from S3, I scheduled a deliberate drill: wipe GitLab, Vault, Harbor’s proxy cache, Authentik’s database, and one Longhorn volume on a Saturday morning, then rebuild everything from Terraform + S3 with a stopwatch running. Total drill time: 4 hours 22 minutes, end to end. About 90 minutes of that was actual rebuild work; the rest was discovering pieces of state I’d accidentally left out of the IaC. ...

TL;DR My Mac Studio M3 Ultra runs Ollama with 70B+ models but isn’t a k3s node. I needed it to show up in Grafana next to the cluster workloads. The solution: node_exporter for system metrics, a Go reverse proxy for per-model inference metrics, a custom Python exporter for model inventory and VRAM tracking, and Grafana Alloy for shipping logs to Loki. All four services managed by Ansible, all metrics scraped by the cluster’s Prometheus. ...

TL;DR Today was cluster genesis. Starting from 3 bare Proxmox hosts, I built the entire infrastructure-as-code pipeline: Terraform to provision VMs from cloud-init templates, Ansible to configure and bootstrap k3s, and a full GitOps deployment model with SOPS-encrypted secrets and S3-backed Terraform state. By end of day: 3 server nodes, 3 agent nodes, cert-manager with Route53 DNS-01 validation, and self-hosted GitHub Actions runners on the cluster itself. The Architecture The design goal was simple: everything as code, nothing manual, everything reproducible. ...