Infrastructure

TL;DR My production homelab runs on Lenovo M920q tinies, and I still think those are the sweet spot. But if I were starting over today with a tight budget, I’d buy a stack of government-surplus Dell OptiPlex 7060 and 7070 desktops instead. They go for around $150 each refurbished — 6-core 8th/9th-gen Intel, an SSD, and Windows 11 already on them — and they make excellent Kubernetes nodes with exactly two cheap upgrades: a bit more RAM and a second network card. ...

TL;DR Stable Diffusion 3.5 Large is 15 GB. RealVisXL is 6.5 GB. Throw in a few LoRAs and a VAE, and your SSD hits the wall fast. I run a MinIO bucket as the long-tail model store, sync it to a local overflow directory on a 30-minute schedule via rclone, and register both the hot (SSD) and cold (synced overflow) paths in ComfyUI’s extra_model_paths.yaml. Models appear transparently; the loader searches both tiers. A fresh model lands in MinIO, appears locally within 30 minutes, and ComfyUI finds it without any manual shuffling. ...

TL;DR A Mac Studio M3 Ultra costs the same as a single 4090 but comes with 256 GB of unified memory and 60-core GPU, all running at 100–200 W under inference. I stopped trying to pass MPS into containers and instead run Ollama and ComfyUI natively on macOS, then proxy them back into k3s as simple Kubernetes Services with manual Endpoints. Two Mac Studios connected via Thunderbolt 5 split the load: one handles hot-path LLM inference and embeddings, the other runs the heavy forge for diffusion and long-horizon reasoning. Both are cheaper to run than a single-socket A100 and require no special driver stacks. ...

TL;DR Purpose-built smart-home wall panels are expensive, locked down, and usually underpowered. A used Microsoft Surface Pro 7 — Core i5 or i7, 16 GB RAM, a sharp 12.3" touchscreen — runs about $150 on the surplus market and makes a fantastic wall-mounted dashboard for Home Assistant, Grafana, or whatever you self-host. It’s a full x86 PC behind a great touchscreen, so it runs a real browser with your real dashboards, not a stripped-down panel app. Here’s the build. ...

TL;DR Consumer mesh Wi-Fi is expensive and locked down. Meanwhile, enterprises retire perfectly good 802.11ac access points by the pallet and dump them on the surplus market for a few dollars each. The Extreme Networks WS-AP3825i — a 3x3 MIMO, dual-band, PoE business AP — runs about $6 used, and it’ll happily run OpenWrt: no vendor controller, no license, no cloud account, just a clean Linux router you own. I bought a pallet of them. Here’s the why, the how, and the one mistake that turns one into a paperweight. ...

TL;DR Everyone reaches for a mini PC or a Pi for a homelab node. The thing nobody tells you: a used business laptop is a server with a built-in UPS, screen, and keyboard bolted on for free. A Dell Latitude 7400 — 8th-gen Core i5, 16 GB RAM, NVMe SSD — runs about $150 used, draws ~10 W with the lid shut, and when the power flickers it doesn’t even notice, because it’s running off its own battery. I run a couple as edge nodes. Here’s the case for it and the five-minute headless setup. ...

TL;DR Over the past two months, I have made a series of stability improvements to my k3s homelab cluster. The biggest wins: migrating from AWS ECR to self-hosted Harbor (eliminating 12-hour token expiry), fixing recurring Grafana crashes caused by SQLite corruption on Longhorn, recovering pve4 after a failed LXC experiment, hardening NetworkPolicies to close gaps in pod-to-host traffic rules, and patching multiple CVEs across the media stack. The cluster now runs 7/7 nodes on k3s v1.34.4, all services monitored, all images pulled from Harbor with static credentials that never expire. ...

TL;DR I am designing a WireGuard VPN mesh to connect a small tech collective – a group of friends who each run their own infrastructure. The topology is hub-and-spoke with my k3s cluster as the hub, connecting 4+ remote sites over encrypted tunnels. Shared services include Jellyfin media federation, distributed CI/CD runners, LAN gaming, and centralized monitoring. The logging pipeline is privacy-first: all log filtering and anonymization happens at the edge (spoke side) before anything ships to the hub. This post covers the network design, the three-layer firewall architecture, the privacy model, and the phased rollout plan. ...

TL;DR I am deploying Authentik as a centralized identity provider for my k3s cluster. It replaces the current OAuth2 Proxy setup with proper SSO, federates Google as a social login source, and introduces group-based RBAC (admins, writers, readers) across all services. The migration is phased – public services first via Traefik forwardAuth, then internal services via native OIDC, then proxy-protected apps that have no OIDC support. OAuth2 Proxy stays in git for instant rollback. This post covers the architecture, the user model, the edge security design, and the gotchas I expect to hit. ...

TL;DR The homelab runs 42 Kubernetes namespaces across 7 nodes (3 control plane, 4 workers) on 4 Lenovo ThinkCentre M920q mini PCs running Proxmox VE. This post is the result of a full infrastructure audit — reconciling what’s actually running against what’s documented, catching version drift, and noting what’s been added, removed, or broken since the last check. Compute Four Lenovo ThinkCentre M920q nodes form the physical layer: Host CPU RAM NVMe Role pve1 i5-8500T 32GB 512GB 1 server VM + 1 agent VM pve2 i5-8500T 32GB 512GB 1 server VM + 1 agent VM pve3 i5-8500T 32GB 512GB 1 server VM + 1 agent VM pve4 i7-8700T 32GB 512GB 1 agent VM (GPU passthrough) The k3s cluster runs v1.34.4+k3s1 with embedded etcd for HA. All 7 nodes report Ready. Server VMs get 2 cores and 6GB each — just enough for etcd and the API server. Agent VMs are beefier: 6 cores and 22GB on pve1-3, 12 cores and 28GB on pve4. ...