Devops

TL;DR OpenClaw, my self-hosted AI trading agent, was running in a fat container with 46 Critical CVEs, no network restrictions, and no automated vulnerability scanning. I fixed all three: multi-stage Dockerfile dropped the CVE count to single digits, default-deny NetworkPolicies locked down traffic, and a daily CronJob triages Trivy scan results via local LLM and posts a digest to Slack. Total cost of the automated triage: $0/day. The problem with AI agent containers AI agent containers are uniquely bad from a security perspective. They need: ...

TL;DR I built an “Ops Dream Worker” — a Kubernetes CronJob that runs at 3 AM, inspects the cluster, identifies improvements, and files GitHub issues with specific fixes. It runs entirely on local models (Mac Studio M3 Ultra), costs $0 per run, and went through 240 A/B test iterations to optimize the prompts. The anti-hallucination patterns were harder to get right than the analysis itself. The idea I have a k3s cluster with ~40 deployed services. I maintain it solo. There’s always something that could be better — a deployment missing resource limits, a CronJob that’s been failing silently, an ingress without SSO protection, a container image with known CVEs. These improvements pile up because I’m usually focused on building features, not auditing infrastructure. ...

TL;DR Goss tests the image, not the running container. Use dgoss run --env-file .env to inject your environment and then assert in three layers: does the var exist, is it non-empty, and does it actually authenticate? That layering tells you exactly where the chain breaks instead of just “MySQL connection failed.” The problem I saw someone in a DevOps forum wrestling with this exact thing. They were manually debugging why their .env values weren’t translating properly into MySQL credentials, and had turned to Goss to automate sanity checks. Two questions tripped them up: ...

TL;DR AWS Lens is an open-source Electron desktop app for managing AWS resources — EC2, S3, Lambda, IAM, Cost Explorer, and more. I wanted it accessible from my browser without running a desktop app. I adapted it to run as a containerized Express server on k3s, fixed a class of runtime crashes from the Electron-to-web adapter, hardened it against three security issues, and deployed it behind Traefik and Let’s Encrypt. The changes are open-source in BoraKostem/AWS-Lens#21. ...

TL;DR Wiki.js 2.x ships Mermaid 8.8.2, released in 2020. Mermaid 11 — the current stable release — adds timeline diagrams, improved gitGraph, better theming, and fixes years of rendering bugs. The upstream project defers this upgrade to Wiki.js v3, which has no release date. The PR queue has sat idle for over a year. I forked requarks/wiki at tag v2.5.312, upgraded Mermaid in-place, patched 8 CVEs including one Critical SAML authentication bypass, reduced the vulnerability count from 8 Critical / 48 High to 3 Critical / 42 High, and deployed it to the cluster. The fork stays close to upstream — Vue 2 and Webpack 4 are untouched. Only the Mermaid surface and security dependencies are modified. ...

TL;DR The blog is deployed by GitHub Actions runners running inside the same k3s cluster it’s talking about. A push to main with content under hugo/ triggers a build, a two-pass S3 sync, and a CloudFront invalidation. A daily 06:00 UTC cron handles future-dated posts so I can commit a backlog and let them drip out on schedule. After every successful deploy, a Playwright job kicks off and scans the live site for broken links, visual regressions, and security header compliance. The whole thing runs on eight self-hosted amd64 runners managed by GitHub’s Actions Runner Controller (ARC) in the cluster. Not a single managed CI minute gets billed. ...

TL;DR On March 21, I shipped meaningful work across five repositories in a single day: a 13,674-line stock trading platform from scratch, a Harbor container registry replacing AWS ECR across 13 CI workflows, API key authentication and an HA proxy for digital signage, inventory sell signals for a trading card tracker, and an OpenClaw cost optimization that killed an idle token burn. Every commit was co-authored with Claude. This post breaks down the mechanics of how that actually works – the prompting patterns, the failure modes, the things I would not let the AI do, and the real throughput multiplier. ...

TL;DR AWS ECR tokens expire every 12 hours. Every time the cron job that refreshes the pull secret fails, image pulls break cluster-wide. Docker Hub’s anonymous rate limit (100 pulls/6 hours) started hitting during CI builds that pull nginx:alpine and python:3.12-slim. I replaced both with self-hosted Harbor for container images and Gitea for package registries (PyPI, npm), backed by NFS on the NAS, deployed via Ansible and Helm, with Trivy vulnerability scanning on push. Thirteen CI workflows were updated in a single commit. Pull secrets never expire. Images never rate-limit. Monthly ECR cost drops to zero. ...

TL;DR AI coding tools are now default behavior for most developers, not an experiment. If you manage a team and you haven’t formalized this, you have ungoverned spend, security exposure, and inconsistent behavior happening right now. The fix isn’t to take the tools away — it’s to pick one, pay for it centrally, encode your policies into the AI itself using instruction files and skills, and govern the control folder rather than individual usage. Here’s the framework I’d implement. ...

TL;DR AI tools have caused multiple production incidents in this cluster. The AI alert responder agent alone generated 14 documented failure patterns before it became reliable. A security scanner deployed by AI applied restricted PodSecurity labels to every namespace, silently blocking pod creation for half the applications in the cluster. The service selector trap – where AI routes 50% of requests to PostgreSQL instead of the application – appeared in 4 separate incidents before guardrails stopped it. This post catalogs the failure patterns, the five-layer guardrail architecture built to prevent them, and an honest assessment of what still goes wrong. ...