Monitoring

TL;DR The screens on eight surplus all-in-ones show an ops board: bids, ship queue, pickups, repair backlog, cluster health, weather and traffic. They sit on desks around the office and do double duty: a workstation when someone sits down, the board the rest of the time. The same machines are on their way to becoming k3s worker nodes (one has joined so far), and that build gets its own post. It’s one static HTML page in Chrome kiosk mode. No framework server, no Kubernetes workload. Ansible deploys it. 15 scenes sit in a weighted registry and rotate off the wall clock, about 34 seconds each, so eight screens agree without talking to each other. When a fetch fails the page keeps the last good data and shows its age. A board with stale data and a warning beats a blank screen. It’s a different animal from my older home signage system, which is a k8s app with Raspberry Pi kiosks. One real thing it did: on 2026-09-06 the home cluster’s daily Longhorn backup had stopped completing, and the board’s home-cluster card showed the critical alert. Not the same project as the home signage I’ve written about digital signage on k3s before: an Angular app, a handful of Flask services, MQTT, Raspberry Pis driving displays. That’s a proper application and it lives in the home cluster. ...

TL;DR The dangerous remote failure isn’t the one that errors. It’s the one that reports success — exit code 0, green tick, desktop loads, nothing in the logs. Five real ones from the last two months, each with the wrong hypothesis I actually held. Your diagnostic tool can be the thing lying to you. A packet capture showed traffic arriving on a port that was being dropped, because capture taps before the firewall does. Provisioning needs sentinel files, not vibes. “It looks finished” is not a completion signal; a file the process writes at the end is. Where agents genuinely help: assembling the four facts a diagnosis needs, and recalling the failure signature from three weeks ago. Where they hurt: confidently narrating a plausible cause for a symptom they can’t see. Every one of these produced a page in the knowledge base with the wrong hypothesis recorded, because that’s the field that saves the next hour. Success is not a signal I run machines I can’t put my hands on: a bench of surplus desktops at an office across town, cluster nodes, a couple of workstations belonging to other people. Remote hands come from a KVM device and SSH, and I’ve written about that hardware setup already. ...

TL;DR Everything in my resale pipeline runs on CronJobs, which is correct until the moment you want fresh data now and the next run is 40 minutes out. Fix: a refresh button on each dashboard that creates a Kubernetes Job from the existing CronJob’s template, via the API, using a dedicated ServiceAccount with a tightly scoped Role. The web app gets permission to create Jobs in one namespace and nothing else. No cluster-admin, no shelling out to kubectl, no shared token. Six dashboards: opportunities, active bids, sold, listings, pickups, contacts. Every table sorts on every column, which is not a nice-to-have — unsorted tables hide your worst-performing categories. The metric that changed my buying wasn’t margin. It was days from sourced to sold, by category — and it says two categories I liked were quietly eating all my working capital. Cron is right until you’re standing in front of the shelf The pipeline is scheduled work. Crawlers every six hours, evaluation hourly, order sync every six hours, pack slips twice an hour. That’s the right architecture: it’s cheap, it’s resilient, it recovers on the next tick, and nothing depends on me being awake. ...

TL;DR Deterministic guardrails run first. PreToolUse regex blocks destructive commands (kubectl delete, terraform destroy, DROP TABLE, rm -rf) before the LLM is consulted. LLM safety is probabilistic; regex is not. Goal-drift detector compares initial objective vs final commit via embedding cosine similarity; alerts if divergence > 40%. Test-deletion sentinel hard-fails CI if tests are deleted without a [test-refactor] tag—you can’t hallucinate past the regex. Cost-spike alerts monitor 24h rolling spend at 80% (warn) and 100% (critical), plus a circuit breaker that opens after 3 failed runs. Action audit flags when the execution plan diverged wildly from the initial strategy—catches scope creep before it becomes expensive. The Problem: Autonomy Without a Leash Last year I started automating routine cluster work—dependency upgrades, CI improvements, minor bug fixes—by running Claude in the background with Git/Terraform access. The first three months were great. The fourth month I woke up to a $180 cloud bill. ...

TL;DR My Mac Studio M3 Ultra runs Ollama with 70B+ models but isn’t a k3s node. I needed it to show up in Grafana next to the cluster workloads. The solution: node_exporter for system metrics, a Go reverse proxy for per-model inference metrics, a custom Python exporter for model inventory and VRAM tracking, and Grafana Alloy for shipping logs to Loki. All four services managed by Ansible, all metrics scraped by the cluster’s Prometheus. ...

TL;DR I am designing a WireGuard VPN mesh to connect a small tech collective – a group of friends who each run their own infrastructure. The topology is hub-and-spoke with my k3s cluster as the hub, connecting 4+ remote sites over encrypted tunnels. Shared services include Jellyfin media federation, distributed CI/CD runners, LAN gaming, and centralized monitoring. The logging pipeline is privacy-first: all log filtering and anonymization happens at the edge (spoke side) before anything ships to the hub. This post covers the network design, the three-layer firewall architecture, the privacy model, and the phased rollout plan. ...

TL;DR The homelab runs 42 Kubernetes namespaces across 7 nodes (3 control plane, 4 workers) on 4 Lenovo ThinkCentre M920q mini PCs running Proxmox VE. This post is the result of a full infrastructure audit — reconciling what’s actually running against what’s documented, catching version drift, and noting what’s been added, removed, or broken since the last check. Compute Four Lenovo ThinkCentre M920q nodes form the physical layer: Host CPU RAM NVMe Role pve1 i5-8500T 32GB 512GB 1 server VM + 1 agent VM pve2 i5-8500T 32GB 512GB 1 server VM + 1 agent VM pve3 i5-8500T 32GB 512GB 1 server VM + 1 agent VM pve4 i7-8700T 32GB 512GB 1 agent VM (GPU passthrough) The k3s cluster runs v1.34.4+k3s1 with embedded etcd for HA. All 7 nodes report Ready. Server VMs get 2 cores and 6GB each — just enough for etcd and the API server. Agent VMs are beefier: 6 cores and 22GB on pve1-3, 12 cores and 28GB on pve4. ...

TL;DR AI tools have caused multiple production incidents in this cluster. The AI alert responder agent alone generated 14 documented failure patterns before it became reliable. A security scanner deployed by AI applied restricted PodSecurity labels to every namespace, silently blocking pod creation for half the applications in the cluster. The service selector trap – where AI routes 50% of requests to PostgreSQL instead of the application – appeared in 4 separate incidents before guardrails stopped it. This post catalogs the failure patterns, the five-layer guardrail architecture built to prevent them, and an honest assessment of what still goes wrong. ...

TL;DR After running the cluster for nearly two weeks, today I took a step back to document and optimize the monitoring stack. This covers kube-prometheus-stack (Prometheus + Grafana + AlertManager), Loki for log aggregation, custom dashboards for every service, alert tuning to reduce noise, and the cluster-wide performance benchmarks I ran to establish baseline metrics. The Monitoring Architecture ┌──────────────────────────────────────────────────┐ │ Grafana │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Metrics │ │ Logs │ │ Alerts │ │ │ │ Explorer │ │ Explorer │ │ Rules │ │ │ └──────┬───┘ └──────┬───┘ └──────┬───┘ │ └─────────┼──────────────┼─────────────┼───────────┘ │ │ │ ┌─────┴─────┐ ┌─────┴─────┐ │ │Prometheus │ │ Loki │ │ │ (metrics) │ │ (logs) │ │ └─────┬─────┘ └─────┬─────┘ │ │ │ ┌─────┴──────┐ ┌──────┴──────┐ ┌─────┴────┐ │AlertManager│ │ Exporters │ │Promtail │ │ → Slack │ │ node │ │(log │ └────────────┘ │ kube-state │ │ shipper) │ │ cAdvisor │ └──────────┘ │ custom │ └─────────────┘ kube-prometheus-stack The foundation is kube-prometheus-stack, deployed via Helm. This single chart installs: ...

TL;DR Today I deployed two significant additions to the cluster: an AI-powered Alert Responder that uses AWS Bedrock (Amazon Nova Micro) to analyze Prometheus alerts and post remediation suggestions to Slack, and a multi-user dev workspace with per-user environments. I also hardened the cluster by constraining all workloads to the correct architecture nodes and fixing arm64 scheduling issues. The Alert Responder Running 13+ applications on a homelab cluster means alerts fire regularly. Most are straightforward — high memory, restart loops, certificate expiry warnings — but analyzing each one, determining root cause, and knowing the right remediation command gets tedious, especially at 2 AM. ...