TL;DR

  • One worker in my home k3s cluster, k3s-agent-4, has been starved for CPU on and off since 2026-06-13. I’ve written about it in my cluster knowledge base more than a dozen times.
  • It was declared fixed at least once and the declaration was falsified twice. “Quiet for a few weeks” turned out to be temporary.
  • What the node showed: requests at 87 percent, limits at 629 percent of CPU, probes timing out, and Longhorn’s engine-image pod restarting hundreds of times.
  • The alert I’d written for this could not fire on a node starved enough to silence its own exporter. Replacing it was the most useful change of the whole saga.
  • The latest and most promising finding is physical: the Proxmox host under that VM clamps its CPU to 800 MHz under load (BD PROCHOT). As of early October that is open, not fixed.
  • This is a post about the shape of a chronic problem, and how many plausible explanations can be true at once.

The cluster in one paragraph

The home cluster is four Proxmox hosts running k3s VMs: a control plane and four workers, with Longhorn for storage. (It used to have three control planes, until I collapsed it to one on 2026-09-05.) Workers have 6 to 10 vCPU each. The worker in this story lives on the fourth host, the only one with hyperthreading: an i7 with six physical cores and twelve threads, hosting the 10-vCPU agent.

My knowledge base for the cluster is an agent-written file that a routine sweep appends to twice a day (I wrote about that setup in the agent-written knowledge base post). This saga is mostly reconstructed from it.

June 13: the first symptom was a cron job

The first sign wasn’t CPU at all. A CronJob that watches Jellyfin failed three days running with no error in the container. The pod was just killed after 45 minutes by activeDeadlineSeconds. Timeouts with no error are always either a network problem or an overloaded node, so I checked the node. k3s-agent-4 was at 95 to 98 percent CPU, running eight Postgres instances, Prometheus, a pile of CronJobs and Longhorn.

Fix 1: a soft node-affinity that prefers anywhere but agent-4, stamped onto 47 CronJob manifests. It’s preferred, not required, so a CronJob can still land there if nothing else fits, and one did.

That wasn’t the end of fix 1. Coverage drifted. On 2026-09-09 a sweep found six CronJobs added or missed afterward. A re-run of the grep that afternoon found 28 more, so roughly half had been covered. They were backfilled, and on 2026-09-21 I made it a CI lint that fails the pipeline if a non-archived CronJob manifest doesn’t mention the node. That closed the coverage problem. It didn’t fix the node.

The second symptom: Longhorn’s own pods

The other recurring signature was Longhorn’s engine-image DaemonSet pod on that node restarting. Its liveness probe runs a small command with a four-second timeout, three failures allowed. On a starved node even that exec runs long three times in a row. The pod gets SIGKILLed (exit 137) and restarts. The restart counter for engine-image passed 600 by late August and 940 by late September, with the CSI plugin over 460, on that node only, against 7 to 12 on peers.

The pod self-heals between restarts, which is why this never looked like an outage. But each failure window risks brief I/O disruption to attached volumes.

I ruled out I/O early. Load average on the node reached 75 to 159, with 130-plus runnable processes, while I/O wait was around zero and the busiest disk’s weighted I/O time peaked at 0.24. That matters because exec probes timing out reads like a disk stall. It was CPU.

What the starved node actually looks like

On 2026-09-14, in the middle of an episode, the kubelet answered a stats request in about a tenth of a second while the node sat at 43 to 62 percent CPU. That’s not saturation. But pressure stall info on the kubelet cgroup read a “full” average of 38 percent and on the pods cgroup a “some” average of 79 percent. Throttling, not raw load. The numbers behind it:

  • CPU requests 87 percent of allocatable on the node
  • CPU limits 629 percent
  • Memory limits 356 percent

When limits are six times what the node has, CFS quota throttling stalls things that are individually well-behaved. kubectl top nodes showed the node as <unknown> during these episodes, because metrics-server couldn’t scrape it. The GitLab HPAs then complained that they couldn’t get metrics. That was pure downstream noise, but it looked like a GitLab problem the first time.

Why that node? It’s organic scheduler drift, not pinning. It had 84 to 105 pods when other workers had 52 to 86, probably because its larger allocatable pool attracts scheduling.

The probe-timeout patches

A pattern showed up: any component whose probe leaves timeoutSeconds at the Kubernetes default of 1 second restarts under contention. I raised the timeout from 1 to 5 on four of them: the registry, the Prometheus operator, a vulnerability scanner, and cert-manager’s webhook. Three of those needed a patch step in the deploy job because the chart or manifest offers no values hook. Two database exec probes got 3 seconds.

These tolerate contention. They don’t fix it, and the knowledge base entry says so. I’m mentioning them because they’re the “silence the symptom” move, and it’s a legitimate move as long as you don’t write it down as a cure.

The alert that couldn’t fire

I’d had NodeCPUSaturation (over 95 percent for 10 minutes) since the first incident. On 2026-08-09 the node sat at 99 percent for 105 minutes and it never fired. Three incidents had produced zero notifications.

The reason is structural. A node that pegged can’t serve its own /metrics. Each missed scrape makes the series stale and restarts the 10-minute for: timer, so ten continuous minutes never accumulate. And the generic “target down for 10 minutes” alert is defeated by the same flap.

The replacement is NodeExporterScrapeFlapping: availability of the node-exporter target over a 30-minute window below 80 percent, for 5 minutes. Backtested over 7 days it had 17 breaching samples, all that node, all inside the incident. Healthy baseline sat between 0.93 and 1.00, and the incident floor was 0.21.

The principle I took from it: ask what a metric does in the failure state. “The exporter went quiet” is evidence, not missing data.

The capacity arithmetic

By late August I had hand-measured free capacity eleven sweeps in a row and deferred the decision each time. On 2026-08-30 I turned it into an alert. To drain node N its requests must fit in everyone else’s free capacity, which works out to: total free ≥ allocatable of N. So any worker is drainable only if total free capacity is at least the smallest worker’s allocatable.

On that day, free CPU was 3.27 cores against a 6-core minimum, and free memory 12.7 GB against 16.7. No worker could be drained. And kubectl drain doesn’t refuse when short of capacity. It evicts anyway and the pods sit Pending, so an ordinary rolling patch would have silently taken services down.

Two details that cost me time. The query has to join requests to running pods, or completed CronJob pods that kube-state-metrics still exports inflate requests by about 24 percent and hide the problem. And control-plane nodes don’t help: they’re tainted and run almost nothing.

Then on 2026-09-05 an agent-1 memory spiral showed what “no headroom” means. Memory fell monotonically over twenty minutes, the node stopped heartbeating, about 53 pods got evicted, and GitLab went down because its Redis replacement was correctly blocked by a Multi-Attach error on the still-terminating old pod. I didn’t force-delete or detach anything. Force-detaching is exactly what Multi-Attach protects against.

False fix 2: consolidation

That same day I finished collapsing three control planes into one. It looked like the structural fix: by 09-08 requests had eased to 15 to 30 percent CPU and 64 to 72 percent memory, and the drain-headroom alerts cleared. The log entry said “resolved by consolidation”.

On 2026-09-14 it came back: metrics-server scrape failures for over an hour, Longhorn manager flapping with “node is down”, drain headroom exhausted, exporter flapping. On 2026-09-28, MetalLB speaker, node-exporter and an NFS provisioner restarted within the same 40 minutes. The log entry is blunt: treat “quiet for weeks” as temporary.

The tempting fix that wasn’t

On 2026-09-11, with one control plane, KubeCPUOvercommit started firing. The HA branch of the rule evaluates requests minus (allocatable minus the largest node’s allocatable). Live, that was 22.325 minus 22, so over by a third of a core.

The tempting fix was adding two cores to the control-plane VM. That makes the denominator bigger and turns the alert green. But the control plane has a NoSchedule taint, so it adds zero schedulable capacity. It would have bought a green dashboard and destroyed the signal. Also, max(allocatable) is in the formula, so growing the largest worker makes it worse: taking agent-4 from 8 to 10 cores added 2 cores of supply and 2 to the max, and they cancelled out.

Actual CPU use across the cluster at that moment was 3.37 cores against 32 allocatable. Requests were 6.6 times real consumption. That alert described a reservation problem, never a saturation problem. Both can be true on one node: reservations in the red on the cluster as a whole, and real starvation on one machine.

The layer under the VM

On 2026-09-27 GitLab pages were slow. Every app-level signal looked plausible and wrong: Redis slowlog entries, Rails logging 20 to 40 ms per query, Puma queue times of 1 to 10 seconds. Yet Postgres was averaging 0.035 cores and nodes showed about 20 percent CPU with no throttling.

What found it was luck. The HPA started a second web pod on a different node, and the same controller action cost 0.38 seconds of CPU there against 3.3 to 3.9 seconds on agent-4. Same code, same page, ten times the CPU. That pointed at the hardware.

The Proxmox host under agent-4 was clamping its CPU to 800 MHz when load arrived. Governor performance, turbo enabled, power limits identical to its siblings. The performance-limit MSR showed PROCHOT asserted; in one 75-second sample it was active in 95 of 150 readings, every core at exactly 800 MHz, 34 to 42 degrees below the thermal limit, with zero thermal-throttle events. The sibling hosts hold about 3.2 GHz. The frequency graph swings between 800 and 3,800 MHz daily, as far back as I have data for (2026-09-17).

That makes every pod on the node run at about a quarter speed under load. Suspects, most likely first: the power adapter (not recognised, or under-rated, so the embedded controller asserts the line), or VRM heat from blocked airflow. A self-reverting power-limit test that would tell them apart hasn’t been run. I’m not going to clear the PROCHOT enable bit, because it’s the board protecting the adapter.

I don’t know whether this explains June. My earliest clock data starts in mid-September, so I can’t prove the clamp was there in June. What I can say is that it’s the first explanation that accounts for all of what the other nodes didn’t show.

What I’d carry out of this

  1. A “resolved” entry needs an expiry. Mine got falsified twice. Now the knowledge-base entry names what would have to change before I accept quiet as fixed.
  2. If an alert depends on the thing it’s alerting about, it’ll go silent exactly then. Alert on availability over a window.
  3. Before resizing anything to make a capacity alert green, check the target’s taints.
  4. When latency is high and backends are idle, compare the same operation across nodes before tuning anything. A tenfold CPU difference between identical pods is a hardware signal.
  5. Patches that raise timeouts are fine. Calling them a fix isn’t.

As of the last check on the early-October knowledge base, the node is still contended, the clamp is still open, and rebalancing pods off that node (topology spread, or a hard anti-affinity for stateless jobs) is a deferred decision. It touches StatefulSets and volumes, and I’d rather do that attended than in a routine sweep.