Infrastructure

TL;DR The blog is deployed by GitHub Actions runners running inside the same k3s cluster it’s talking about. A push to main with content under hugo/ triggers a build, a two-pass S3 sync, and a CloudFront invalidation. A daily 06:00 UTC cron handles future-dated posts so I can commit a backlog and let them drip out on schedule. After every successful deploy, a Playwright job kicks off and scans the live site for broken links, visual regressions, and security header compliance. The whole thing runs on eight self-hosted amd64 runners managed by GitHub’s Actions Runner Controller (ARC) in the cluster. Not a single managed CI minute gets billed. ...

TL;DR I spent time evaluating Linkerd — the CNCF graduated service mesh — for my homelab k3s cluster. The conclusion: it’s an impressive piece of engineering with genuinely useful features like automatic mTLS, post-quantum cryptography, and per-service observability. But for a cluster with ~20 workloads and a single operator, the operational overhead outweighs the benefits today. I’ve written a complete deployment plan so I can adopt it quickly when the cluster grows to the point where it makes sense. ...

TL;DR AWS ECR tokens expire every 12 hours. Every time the cron job that refreshes the pull secret fails, image pulls break cluster-wide. Docker Hub’s anonymous rate limit (100 pulls/6 hours) started hitting during CI builds that pull nginx:alpine and python:3.12-slim. I replaced both with self-hosted Harbor for container images and Gitea for package registries (PyPI, npm), backed by NFS on the NAS, deployed via Ansible and Helm, with Trivy vulnerability scanning on push. Thirteen CI workflows were updated in a single commit. Pull secrets never expire. Images never rate-limit. Monthly ECR cost drops to zero. ...

TL;DR PETG is the go-to filament for functional homelab parts — heat resistant, mechanically strong, and chemically stable. The advertised temperature ranges on most PETG spools (230-250C nozzle) are too low. After systematic testing, the settings that actually produce strong, well-bonded prints on the Bambu Lab P1S are 265C nozzle and 80C bed, with first layer at 270C/85C. These temperatures are 15-35C hotter than what the manufacturer label suggests. Amazon came through with a 4KG bulk delivery that arrived the same day the previous spool ran out, which was cutting it closer than I would like. ...

TL;DR I added a Bambu Lab P1S to the homelab and it has become one of the highest-value additions to the setup. Print quality out of the box is near injection-mold level for functional parts. I have already printed ventilated node enclosures, SFP+ cable routing brackets, custom rack shelves, and equipment mounts. Setup took under 30 minutes from unboxing to the first print. The ability to prototype custom hardware solutions in hours instead of waiting days for shipped parts changes how you approach infrastructure problems. ...

TL;DR This is the moment everything was built for. Three phases of preparation — PostgreSQL provider (Day 3), storage migration (Day 4), state externalization (Day 5) — all leading to a single kubectl scale command. This post covers Phase 4: scaling the Jellyfin StatefulSet to 2 replicas, configuring anti-affinity to spread pods across nodes, running six structured failover tests, building Prometheus alerts, and one test that only partially passed. The headline result: killing a pod causes zero service downtime — users on the surviving replica experience no interruption at all, and displaced users reconnect within seconds. ...

TL;DR Phase 2 is the scariest phase. It’s where we take a running Jellyfin instance with years of playback history, user preferences, and media metadata — then swap the database from SQLite to PostgreSQL and restructure every volume. One wrong move and the family discovers their “Continue Watching” list is gone. This post covers deploying PostgreSQL as a k3s StatefulSet, restructuring Jellyfin’s volume layout from a monolithic RWO PVC to NFS shared config + Longhorn per-pod storage, and building a SQLite-to-PostgreSQL migration tool. ...

TL;DR Prometheus and Grafana both crashed with I/O errors on the same node. Before assuming software, I ran a full hardware audit across all four Proxmox hosts — SMART health, NVMe disk benchmarks (fio), CPU benchmarks (sysbench), memory bandwidth tests, and 10GbE network throughput (iperf3). The result: all hardware is healthy. The I/O errors were Longhorn CSI virtual block device corruption, not physical disk failure. Along the way, I established baseline performance numbers for every subsystem and discovered that custom cooling makes a dramatic difference in thermal performance. ...

TL;DR I upgraded the inter-node networking from 1GbE to 10GbE using Mellanox ConnectX-3 NICs from eBay (~$15-20 each). Two of the three NICs worked immediately. The third needed a firmware flash from 2.33.5220 to 2.42.5000 using mstflint, which required two cold boots to recover from a partially bricked state. All three Proxmox hosts now have 10GbE connectivity via active-backup bonds. I also deployed Security Scanner infrastructure and fixed ARC runner Pod Security Standards. ...

TL;DR Intel iGPU passthrough from Proxmox to a k3s VM enables hardware video transcoding with minimal CPU overhead. This guide covers the complete process: enabling IOMMU, configuring VFIO, blacklisting the i915 driver, rebuilding the VM with q35/OVMF, and verifying VA-API inside the VM. The Intel UHD 630 in the M920q handles H.264 and HEVC encode/decode at real-time speeds. ...