TL;DR

  • The dangerous remote failure isn’t the one that errors. It’s the one that reports success — exit code 0, green tick, desktop loads, nothing in the logs.
  • Five real ones from the last two months, each with the wrong hypothesis I actually held.
  • Your diagnostic tool can be the thing lying to you. A packet capture showed traffic arriving on a port that was being dropped, because capture taps before the firewall does.
  • Provisioning needs sentinel files, not vibes. “It looks finished” is not a completion signal; a file the process writes at the end is.
  • Where agents genuinely help: assembling the four facts a diagnosis needs, and recalling the failure signature from three weeks ago. Where they hurt: confidently narrating a plausible cause for a symptom they can’t see.
  • Every one of these produced a page in the knowledge base with the wrong hypothesis recorded, because that’s the field that saves the next hour.

Success is not a signal

I run machines I can’t put my hands on: a bench of surplus desktops at an office across town, cluster nodes, a couple of workstations belonging to other people. Remote hands come from a KVM device and SSH, and I’ve written about that hardware setup already.

This post isn’t about the tooling. It’s about a specific category of bug that remote work produces far more of than local work does: the failure that reports success.

Locally, you notice. You’re standing there, something’s off, you poke it. Remotely, your entire picture of the machine comes through a handful of narrow channels — a log line, an exit code, a screenshot, a check-in POST — and if all of those say “fine”, the machine is fine as far as you know. For weeks, if nobody looks.

Here are five that got me. In each case I’ve included the hypothesis I actually held, because that’s the part worth reading.

1. The packet capture that lied

Symptom. Machines network-booting on the office bench never got an address. No boot, no errors on the server, nothing.

What I believed. The boot server was misconfigured — wrong subnet declaration, wrong option set, something in the config. I read that config maybe six times.

Why I believed it. I ran a packet capture on the server’s interface and the requests were arriving. Clear as day, the right packets on the right port. Traffic was reaching the box, so the network was fine, so it had to be the service configuration.

Actual cause. The host firewall was dropping them. The default-deny policy had no rule for the boot ports, and the requests come from an unassigned source address (the client has no IP yet), so a rule scoped to the local subnet wouldn’t match anyway.

The lesson that generalises. Packet capture taps the interface before the firewall processes the packet. A dropped packet still shows up in the capture, looking exactly like an accepted one. My primary diagnostic instrument was structurally incapable of distinguishing “arrived and was accepted” from “arrived and was destroyed”, and I trusted it for hours.

If a service isn’t seeing traffic that a capture shows arriving, check the firewall counters, not the capture. That bench had never once served a real client — the whole path had been broken since it was built, and the capture had been reassuring me the entire time.

2. The script that ran without privileges

Symptom. Machines finished provisioning, the finalise step ran, the operator clicked through it, everything looked done. Weeks later I found units in the field still carrying the temporary provisioning account.

What I believed. The cleanup step had a bug in its account-removal logic.

Actual cause. The finalise script was being launched from a shortcut that didn’t request elevation. Under the OS’s privilege model it ran with a filtered token: it started, executed, skipped every operation that needed administrator rights, and exited zero. No error dialog, no log entry, no indication whatsoever that half of it hadn’t happened.

Fix. The script now checks its own privilege level as the very first thing it does and hard-stops with a full-screen red banner if it isn’t elevated. Not a warning — a refusal.

The lesson. A script that needs privilege must assert it has privilege, loudly, before doing anything. Otherwise a partial run is indistinguishable from a complete one, and the person clicking the shortcut has no way to know. This is the same class of bug as a CI job that passes because the tool it was meant to run wasn’t installed.

3. The global flag that made machines wipe themselves forever

Symptom. A wipe-and-reimage run for three machines produced 48 network-boot attempts and 43 image downloads. The bench churned for hours.

What I believed. Flaky hardware or a network problem causing retries.

Actual cause. There’s a force-wipe flag that overrides the “don’t re-wipe a machine that’s already provisioned” guard. It’s a global flag — a file on the server, not a per-machine setting. So: machine finishes imaging, reboots, network-boots as configured, sees the global force-wipe flag still set, and wipes itself again. Forever. Three machines in an infinite loop, each run looking like a perfectly successful deployment.

Fix. Disarm the flag the moment the first wipe starts. Better fix, which I’ve since made: the flag carries a list of machine identifiers rather than being a global boolean, and it self-expires.

The lesson. A global override that persists across the operation it was enabling is a loop generator. If you have a “yes, really, do the dangerous thing” flag, scope it to the specific target and give it a lifetime. And be suspicious of any operation whose per-unit count exceeds the number of units — that ratio is the cheapest loop detector there is.

4. The unit I nearly reimaged for no reason

Symptom. A machine failing to boot with a missing-boot-loader error. Classic dead-install symptom.

What I believed. Imaging failed partway through; reimage it.

Actual cause. The install was complete and fine. The boot loader on the system partition was broken. The main partition had the full OS tree and the sentinel file that the provisioning process writes when it finishes successfully.

Fix. Repair the boot loader — assign the system partition a drive letter, rebuild the boot files, done in two minutes. No reimage.

The lesson. Before you re-run an expensive operation, check whether it actually failed. Sentinel files are what make that check possible: a file written at the end of provisioning is proof the provisioning completed, and it survives a broken boot loader. I now have three of them at different stages, and “which sentinels exist?” is the first question in the runbook. It converts “this machine is broken, start over” into “this machine finished, one component is broken, fix that component.”

5. The boot order that ate itself

Symptom. A freshly-wiped machine cycling endlessly: network boot, wipe, reboot, network boot.

Cause. I’d set the network interface first in the persistent boot order so the machine would pick up its deployment. Which it does — and then does again on every subsequent boot, because a freshly-wiped disk has nothing to boot from and the network is still first in line.

Fix. Set the network interface as a one-time boot override rather than a persistent reorder. The machine boots from the network once, deploys, and reverts to disk-first on its own.

The lesson. For any bootstrap step, prefer one-time overrides to persistent configuration changes. Persistent config that triggers a bootstrap is a bootstrap that runs forever.

Where agents actually help

Three of those five, an agent materially shortened. Not by diagnosing — by doing the legwork a diagnosis needs.

Assembling the facts. A diagnosis usually needs four or five things: the exact error string, what the config currently says, what changed recently, and whether this has happened before. Gathering those takes twenty minutes of context-switching across a KVM screenshot, an SSH session, a git log, and a wiki search. An agent does it in one and hands me a summary. I’m still the one who has the idea.

Recall. This is the underrated one. “Have I seen this before?” is a question I answer badly and an agent answers well, because the knowledge base has the lesson pages and the agent will actually search them. The boot-loader misdiagnosis took two minutes the second time because the recognition signature was written down.

Scripted state changes instead of clicking through firmware menus. Setting boot order, one-time boot targets, and firmware options from a command line rather than driving a BIOS screen through a KVM. The KVM is for when there’s no OS and nothing else works; anything scriptable should be scripted, because a KVM screenshot is a photograph of a claim and a command’s output is the claim itself.

Helping somebody else’s session. There’s a bridge between my agent and a collaborator’s, over a chat channel — I wrote about how that works. For troubleshooting specifically it’s better than I expected, because the useful unit of help is rarely “here’s the fix”, it’s “here are the three things to check and what each result would mean.” That’s a message, not a session.

Where agents hurt

Blunt version: an agent will produce a confident, well-structured, plausible cause for a symptom it cannot observe, and remote troubleshooting is exactly the situation where it can’t observe much.

Every one of the five failures above had an available plausible story. Boot server misconfigured. Cleanup logic bug. Flaky hardware. Failed image. All coherent, all wrong, and all the kind of thing a model will happily elaborate on for four paragraphs if you let it.

So the working rule is: the agent may hold a hypothesis only alongside the specific observation that would falsify it. Not “the firewall is probably dropping it” but “if the firewall is dropping it, the counters on that rule will be non-zero — here’s the command.” That single reframing is what makes agent-assisted diagnosis useful rather than a fluent generator of dead ends.

The corollary is that I don’t ask “what’s wrong with this machine?” I ask “what would distinguish these three causes?” The second question is one a model is genuinely good at, because it’s about the structure of the system rather than the state of a box neither of us can see.

Gotchas

Check your instrument’s position in the pipeline. Capture before the firewall, logs before the buffer flush, a screenshot before the screen repaints. Every diagnostic tool observes at a specific point and is blind to everything after it.

Assert privilege, assert prerequisites, assert the tool exists. Anything that can silently skip work must refuse to start instead. A step that passes because it did nothing is worse than a failure.

Sentinel files at every stage. Cheap, boring, and the difference between “reimage it” and “fix the one broken thing”.

Scope your dangerous overrides to a target and a lifetime. A global “do the destructive thing” flag will eventually be left on.

Count operations per unit. More boots than machines, more downloads than installs, more prints than orders — ratios above one are your cheapest loop detector, and nothing else was going to tell me.

Where this sits

Five failures, all of which reported success, across two months. The tooling didn’t change much. What changed is that provisioning now writes sentinels, dangerous flags are scoped, privilege gets asserted before work begins, and every one of these produced a lesson page with its wrong hypothesis recorded.

That last field is the one I’d keep. The symptom-to-cause mapping is something I could have looked up. The knowledge that my packet capture couldn’t see the firewall is something I could only get by having been wrong for an afternoon, and writing it down is the only way that afternoon buys me anything.