Homelab

TL;DR The digital signage system was pulling weather from OpenWeatherMap, calendar events from Google Calendar, and device status from MQTT – three separate API keys, three separate failure modes. Home Assistant already had all of this data. I built an HA proxy service that exposes weather, forecasts, calendar events, temperature sensors, and arbitrary entity queries through a single Flask API backed by the Home Assistant REST API. Five new endpoints replaced three external dependencies. I also added API key authentication with role-based access control, wrote 37 tests, fixed MQTT addressing after a VLAN migration, and fought through 6 CI/CD fixes to get the pipeline deploying on self-hosted ARC runners. ...

TL;DR GitHub Copilot is more capable than most people give it credit for. I used it heavily – not just for autocomplete, but for multi-file edits, chat-driven debugging, and workspace-aware refactoring. After a year of intensive Copilot usage and a month with Claude Code Max ($100/month for the Max plan with Opus), I moved my primary workflow to Claude Code for infrastructure and backend work. The reason is not that Copilot cannot do these things – it is that Claude Code is faster and I can hand it a task and let it run without babysitting. Copilot still wins for inline code completion in the editor. Claude Code wins when I want to describe a goal and walk away while it executes. ...

TL;DR AWS ECR tokens expire every 12 hours. Every time the cron job that refreshes the pull secret fails, image pulls break cluster-wide. Docker Hub’s anonymous rate limit (100 pulls/6 hours) started hitting during CI builds that pull nginx:alpine and python:3.12-slim. I replaced both with self-hosted Harbor for container images and Gitea for package registries (PyPI, npm), backed by NFS on the NAS, deployed via Ansible and Helm, with Trivy vulnerability scanning on push. Thirteen CI workflows were updated in a single commit. Pull secrets never expire. Images never rate-limit. Monthly ECR cost drops to zero. ...

TL;DR One month ago, I had three empty Lenovo ThinkCentre M920q mini PCs and a Proxmox installer USB. Today, the cluster runs 8 Kubernetes nodes, 15+ applications, full observability with Prometheus and Grafana, AI-powered alert analysis, self-hosted CI/CD, 10GbE networking, and a 3D printer fabricating custom hardware. Total hardware cost: under $800. This post traces the entire journey, day by day, including the things that went wrong. ...

TL;DR PETG is the go-to filament for functional homelab parts — heat resistant, mechanically strong, and chemically stable. The advertised temperature ranges on most PETG spools (230-250C nozzle) are too low. After systematic testing, the settings that actually produce strong, well-bonded prints on the Bambu Lab P1S are 265C nozzle and 80C bed, with first layer at 270C/85C. These temperatures are 15-35C hotter than what the manufacturer label suggests. Amazon came through with a 4KG bulk delivery that arrived the same day the previous spool ran out, which was cutting it closer than I would like. ...

TL;DR I added a Bambu Lab P1S to the homelab and it has become one of the highest-value additions to the setup. Print quality out of the box is near injection-mold level for functional parts. I have already printed ventilated node enclosures, SFP+ cable routing brackets, custom rack shelves, and equipment mounts. Setup took under 30 minutes from unboxing to the first print. The ability to prototype custom hardware solutions in hours instead of waiting days for shipped parts changes how you approach infrastructure problems. ...

TL;DR Over the last six posts, I’ve documented converting Jellyfin from a single-process media server into a two-replica, PostgreSQL-backed, sticky-session-coordinated deployment on k3s. Five of six failover tests passed cleanly. The key result: zero-downtime failover — killing a pod doesn’t take down the service. Users on the surviving replica see no interruption; displaced users reconnect in seconds. Node maintenance no longer kills Jellyfin for the household. But this project isn’t finished, and some problems can’t be solved with this architecture. This final post is an honest inventory of what’s still broken, what was deferred, and what the path forward looks like. ...

TL;DR This is the moment everything was built for. Three phases of preparation — PostgreSQL provider (Day 3), storage migration (Day 4), state externalization (Day 5) — all leading to a single kubectl scale command. This post covers Phase 4: scaling the Jellyfin StatefulSet to 2 replicas, configuring anti-affinity to spread pods across nodes, running six structured failover tests, building Prometheus alerts, and one test that only partially passed. The headline result: killing a pod causes zero service downtime — users on the surviving replica experience no interruption at all, and displaced users reconnect within seconds. ...

TL;DR Phase 3 is where the rubber meets the road. We have PostgreSQL for persistent data (Day 4) and NFS for shared config. But Jellyfin still holds critical runtime state — sessions, users, devices, tasks — in 11 ConcurrentDictionary instances scattered across singleton managers. Two pods with independent memory spaces means two independent views of reality. This post covers the state externalization decision: what got moved to Redis, what got solved by sticky sessions, what got disabled entirely, and why pragmatism beat perfection for a homelab media server. ...

TL;DR Phase 2 is the scariest phase. It’s where we take a running Jellyfin instance with years of playback history, user preferences, and media metadata — then swap the database from SQLite to PostgreSQL and restructure every volume. One wrong move and the family discovers their “Continue Watching” list is gone. This post covers deploying PostgreSQL as a k3s StatefulSet, restructuring Jellyfin’s volume layout from a monolithic RWO PVC to NFS shared config + Longhorn per-pod storage, and building a SQLite-to-PostgreSQL migration tool. ...