Ansible

TL;DR Three weeks after accidentally wiping GitLab with a misdirected blkdiscard and rebuilding from S3, I scheduled a deliberate drill: wipe GitLab, Vault, Harbor’s proxy cache, Authentik’s database, and one Longhorn volume on a Saturday morning, then rebuild everything from Terraform + S3 with a stopwatch running. Total drill time: 4 hours 22 minutes, end to end. About 90 minutes of that was actual rebuild work; the rest was discovering pieces of state I’d accidentally left out of the IaC. ...

TL;DR My Mac Studio M3 Ultra runs Ollama with 70B+ models but isn’t a k3s node. I needed it to show up in Grafana next to the cluster workloads. The solution: node_exporter for system metrics, a Go reverse proxy for per-model inference metrics, a custom Python exporter for model inventory and VRAM tracking, and Grafana Alloy for shipping logs to Loki. All four services managed by Ansible, all metrics scraped by the cluster’s Prometheus. ...

TL;DR Today was cluster genesis. Starting from 3 bare Proxmox hosts, I built the entire infrastructure-as-code pipeline: Terraform to provision VMs from cloud-init templates, Ansible to configure and bootstrap k3s, and a full GitOps deployment model with SOPS-encrypted secrets and S3-backed Terraform state. By end of day: 3 server nodes, 3 agent nodes, cert-manager with Route53 DNS-01 validation, and self-hosted GitHub Actions runners on the cluster itself. The Architecture The design goal was simple: everything as code, nothing manual, everything reproducible. ...