Grading Seven AI Models on Blue-Team Security Work in a VM With No Network Card

TL;DR

I wanted to know which models I can trust for the defensive side of running a homelab: reviewing code for vulnerabilities, reviewing Kubernetes and Terraform config, triaging logs, and reading threat intel without making things up. So I built a harness with 112 frozen synthetic cases, graded everything deterministically (no LLM judge), and ran every model-written fix inside a Proxmox VM whose network card was deleted at the hypervisor.

This is an evaluation for choosing my own deployment, not a reproducible public benchmark. The method is at the bottom, including the parts that are weaker than I’d like.

• Hosted GLM-5.3-Flash had the best pilot score (0.967 composite, two runs), at a median of 8 seconds per case and pennies per pass.
• My local GLM-5.3-Flash (IQ4_XS on the Mac Studio) scored 0.932 on the pilot and 0.959 on the holdout. Hosted Flash led on the pilot, mostly on code fixes. Local led on the smaller holdout. These runs don’t establish a stable quality ordering between the two deployments.
• The two cyber-branded models I tested came last. Vinci-Cyber-123B flagged problems in 8 of 19 clean controls. Foundation-Sec-8B scored 0.296.
• Zero refusals across every model, local or hosted, on every defensive task.
• Most of the “model problems” I hit were configuration problems: token budgets, chat template defaults, provider rate limits, tool-routing filters.

Why I bothered

I already lean on models for MR review and incident digging in the cluster. What I didn’t have was evidence. Every vendor posts cyber benchmark numbers, and they are all self-reported. I also had a specific question: does the quantized GLM I run locally behave differently from the hosted one, and if so, why?

The setup

Four categories, defensive work only:

The sets were frozen by sha256 manifest before any candidate model saw them: an 80-case pilot, a 24-case holdout, and 8 multi-file agent tasks with read-only tools. Code, infra, and logs include clean controls (19 in the pilot), because a reviewer that flags everything is useless.

The harness refuses to run if the manifest changes, the prompt builder can’t import the ground truth, every model gets the same messages per case, and hosted rows pin one provider endpoint with fallbacks off. Refusals, API blocks, truncations, parse errors, and timeouts are reported separately from wrong answers. No model grades anything.

The sandbox. Model-written code runs in a small Proxmox VM. It boots once to install python3 and the guest agent, then the NIC and cloud-init drive are deleted at the hypervisor. Control goes through qm guest exec. Tests run as an unprivileged user, and the VM resets to a clean snapshot between cases. I verified the isolation from both sides: zero NICs on the host side, only lo in the guest, and connection attempts to the gateway, the k3s API, a public resolver, and cloud metadata all fail with “Network is unreachable”.

The VM protects my network. It doesn’t make the tests complete. A fix that passes means it blocked that exploit test and kept those functional tests green, not that the code is now secure in general.

The results

Pilot composite scores, 0 to 1 (the formula is in the appendix). “Runs” is how many full passes each row got.

The last column counts clean controls that returned a gradable answer. An answer that couldn’t be graded scores 0 but isn’t counted as a false alarm, which is why some denominators are below 19 per run.

The 24-case holdout, run only for finalists:

Hosted Flash went from first on the pilot to sixth of eight here, mostly because its holdout working-fix rate was 0.75 while local GLM and full GLM-5.3 fixed everything they were given. Vinci isn’t in this table: it missed the finalist cut on false alarms, and I skipped its holdout. Foundation-Sec’s holdout ran before the finalist rule existed, so it’s shown for context.

What the numbers do and don’t say

Run it twice before you believe it

The three GLM rows got two identical pilot passes each. Hosted Flash scored 0.978 and 0.956, changing on 23 of 80 cases, two of them by 0.25 or more. Full GLM-5.3 scored 0.967 and 0.914, with 27 cases changing and eight big swings. Local GLM scored 0.921 and 0.942, with 28 cases changing and six big swings.

At this level, one run doesn’t rank close models, and two runs barely do. The other rows got one run, so their scores are less settled than the GLM rows’. The pilot separates strong models from weak ones well. It does not rank 0.93 against 0.96.

Local vs hosted is a comparison of deployments

Local and hosted Flash differ in precision (IQ4_XS vs FP8), inference engine, chat template, and whatever the provider does on its side. So this compares two deployments, not quantization alone. No BF16 endpoint existed for either GLM model, so hosted FP8 isn’t an unquantized reference either.

On the pilot, local minus hosted was −0.035 (95% bootstrap interval −0.071 to −0.004). Hosted won 24 cases and local won 15, with 41 ties. Most of the gap was code (−0.102, interval −0.196 to −0.013). On logs and threat intel I found no clear difference (−0.001 and −0.010, intervals spanning or touching zero). That’s not the same as equivalent; I never set an acceptable-difference margin.

On the holdout the direction flipped: local minus hosted was +0.057 (interval +0.004 to +0.124), seven wins to three.

The two statistics measure different things. The bootstrap interval is about the average size of the difference. The sign test only counts who won each case. Neither result is clear by the sign test (p = 0.20 on the pilot, 0.34 on the holdout), and the two sets point opposite ways on the mean. My honest summary: hosted Flash looked better at writing working code fixes on the pilot, and I can’t call an overall winner.

A few suspects did clear themselves:

• Chat template: I diffed the local GLM template against upstream. The 49 differing lines are syntax ports and loop short-circuits, nothing that would change behavior.
• Reasoning effort: max effort didn’t help either side. Hosted went 0.967 → 0.960, and local went 0.932 → 0.902 with three answers truncated.
• Provider: a second FP8 provider for the same Flash checkpoint scored 0.955 vs 0.976 on the same 40 cases, interval touching zero.
• Quantization, for Qwen only: Qwen3.8 27B at BF16 vs Q4_K_M differed by +0.006 with an interval spanning zero. That says nothing direct about GLM.

The two cyber-branded models underperformed

Vinci-Cyber-123B, a LoRA merged onto a 123B coding model, flagged 8 of 19 clean controls and scored 0.713 on the agent track. Foundation-Sec-8B returned unparseable output on 16 of 80 pilot cases, which is a real deployment failure. It wasn’t only formatting, though: on the answers that did parse, it averaged 0.377. Its official GGUF also ships with no chat template, so llama.cpp silently falls back to ChatML unless you supply the publisher’s template. I did.

feyospace-v1-small, a Qwen3.8 27B post-train from my specialist shortlist, scored about the same as its base model on the pilot (+0.004) and lower on the holdout (0.870 vs 0.942). It also declares no license, which rules it out for anything beyond internal testing.

That’s three specific models, not a verdict on security fine-tuning in general. For these, a strong general model was the better choice.

Speed is where hosted clearly wins

These times are the sum of a case’s request durations: prompt processing, reasoning, and generation. They exclude time spent waiting for the shared local server to be free. The other local rows ran four requests at a time on one server during their quality passes, so their latencies aren’t comparable and I’ve left them out.

To estimate what a deadline would do, I took the finished runs and counted any case slower than the budget as a 0. These weren’t separate deadline-limited runs. At 60 seconds per case, local GLM drops to 0.404 and hosted Flash holds at 0.954. Local needs about 15 minutes per case to reach its full score. For live triage, the tail matters as much as the median, and local’s p90 is close to four minutes.

Hosted cost was trivial. One 80-case pass on hosted Flash billed $0.144, and full GLM-5.3 was $1.29. At last reconciliation, about 670 hosted calls had cost $3.93 in total.

Most “model failures” were config failures

1. Empty answers from local GLM. GLM-5.3 always thinks. With a 32-token output budget at default max effort, it spent all 32 tokens reasoning and got cut off. With 256 tokens it answered fine.
2. Qwen’s template defaults to xhigh effort. The Qwen GGUF chat template injects “Reasoning effort is set to xhigh” unless told otherwise. Qwen3.8-Flash-Next truncated 9 of its first 39 answers at the 16,384-token cap. At medium effort it completed all 80.
3. Provider rate limits. One hosted Flash provider returned HTTP 429 on 36 of 40 requests, even after backing off for up to 240 seconds.
4. Forced tool choice. Two hosted providers rejected tool_choice: required and forced named functions with HTTP 404 at the router’s tool-compatibility filter, before the model saw the request. tool_choice: auto worked everywhere, and local llama.cpp handled all three.
5. Weights getting paged out. While I downloaded about 185 GB of other models, the GLM server’s resident memory fell from about 149 GB to 62.6 GB, because llama.cpp mmaps weights without --mlock. Performance numbers come from a separate clean pass.

And the refusal count, across every row and every track: zero.

What I’m doing with this

• Hosted GLM-5.3-Flash for interactive review and triage, where time matters and the inputs are synthetic or sanitized.
• Local GLM-5.3-Flash for anything I don’t want leaving the house, and for overnight batch review.
• Every model-written fix runs in the no-NIC VM before I believe it, from any model.
• None of the security-branded models I tested, until one beats a general model on my own fixtures.

If you build something like this: freeze the fixtures before the first call, include clean controls, pin the provider, check template defaults, and give every finalist the same number of runs. I didn’t do that last one, and it’s the first thing I’d fix.

Appendix: method

How a case is scored

Every case gets a deterministic score from 0 to 1. A row’s overall score is the mean over cases, with repeat runs averaged per case first. Any answer that isn’t a clean completion (refusal, API block, truncation, parse error, timeout, empty) scores 0. So the overall number is a weighted composite, not an accuracy or a fix rate. Those are reported separately.

• Code and infra, planted fault: 0.5 × finding F1 + 0.5 × working fix. A finding counts when its label is in the accepted CWE or rule set, it names the right file, and its line is within ±3 of the sink or decision point. A working fix blocks the exploit or misconfiguration check and passes the functional check.
• Code and infra, clean control: 1 only if the verdict is clean, there’s no scored finding, and nothing regresses. Otherwise 0, and it counts as a flagged clean control.
• Logs, incident: 0.35 finding F1 (label plus a decisive cited line) + 0.2 entity F1 (attacker IPs, compromised accounts and hosts) + 0.1 first malicious event + 0.15 response actions (required set covered, nothing forbidden) + 0.2 citation precision.
• Logs, benign: 1 only with no incident, no findings, and no forbidden actions.
• Threat intel: 0.3 IOC F1 + 0.1 CVE/product + 0.15 ATT&CK technique F1 + 0.15 attribution restraint + 0.3 claim-verdict accuracy. An item counts only with a citation to the line that supports it. An uncited oracle answer has to score below 0.6, and I checked that it does.

Category scores are plain means within the category, and the overall is the mean over all 80 cases. With 20 cases per category, that’s an equal weighting by category.

Intervals are percentile bootstraps over cases (2,000 resamples). Paired comparisons bootstrap the per-case differences, after averaging repeats within each row. The sign test is exact and two-sided, with ties (cases where both rows scored the same) excluded.

Run matrix

Timeline (US Eastern)

• Oct 9 to just after midnight: fixtures written, validated, and frozen by sha256 before any candidate call. Prompt version, output budget (16,384 tokens; 8,192 for Foundation-Sec’s smaller context), and sampling (temperature 0.6, top-p 0.95) fixed.
• Oct 10, small hours: the first hosted batch, including the first hosted GLM holdout runs. The holdout was opened here, early. Grader g2 came in during this batch, after the first graded run and before any cross-model comparison.
• 09:40: Qwen-architecture rows switched from the template’s xhigh default to medium effort. GLM rows stayed at high. This is a configuration difference between families, not a matched setting.
• Mid-morning: grader g4, from an outside review.
• 10:25: finalist rule frozen, before most non-GLM pilot results existed. GLM rows were finalists by design. Any other row qualified if its pilot score was within 0.10 of the best GLM row, or it was the best non-GLM local row at 0.80 or above, and it flagged no more than 15% of clean controls and it completed at least 90% of cases.
• 21:30: grader g5. 21:45: Flash-Next effort corrected and its pilot rerun.
• Oct 11 to 12: remaining finalist holdouts and repair runs, then a final regrade of every stored answer and the report.

The last grader revisions came after the hosted holdout answers existed. As far as my records show, none was prompted by a holdout result, and none of the three fixture defects below is a holdout case. But the reviews weren’t blind to which model wrote what, so the holdout is less pristine than “sealed” suggests.

Grader history and sensitivity

1. g1: frozen with the fixtures.
2. g2: the prompt asks for the sink or decision line, so either is accepted. Neutral observations match by label and file. Line-range citations no longer count as invented.
3. g3: when two findings with the same label share a file, they’re assigned by distance.
4. g4: threat-intel items need a supporting citation.
5. g5: log account names match across domain qualification (CORP\tnguyen equals tnguyen).

Every revision was applied to every stored answer, with no new inference, and the report generator refuses to mix grader versions. A review of disputed cases across all rows found three fixture defects. In each, a defensible answer was marked wrong. Excluding those three cases moves things slightly: hosted Flash 0.972, full GLM-5.3 0.945, Flash-Next 0.945, local Flash 0.936, feyospace 0.923, Qwen BF16 0.920, Qwen Q4 0.900, Vinci 0.749, Foundation-Sec 0.302. Local Flash and Flash-Next swap places. I haven’t published a full ranking under each grader version.

Treat that as a caveat: five versions is enough revision that the metric could have drifted toward the outputs I happened to read.

Where the cases came from

All 112 cases were written by four Claude-based authoring agents, one per category, from a fixed spec. Claude wasn’t a candidate, and no candidate model helped write or check them. Each case was validated mechanically: the original fails its exploit or misconfiguration check and keeps working, the reference fix passes both, clean controls pass both, an oracle answer scores 1.0, and empty or wrong answers fail.

The pilot and holdout came from the same authors and spec at the same time, so they share style and probably templates. The manifest proves they weren’t changed later. It doesn’t prove they’re independent of each other, or representative of real incidents, which are messier.

What the grader can and can’t recognize

The grader accepts some legitimate variation: either the sink or the decision line, ±3 lines, several accepted CWE or rule labels per finding, and listed extra observations. One SSRF case had its hidden tests changed before the freeze so a resolve-then-check fix passes offline, and I verified that with an independently written fix. But I didn’t test alternative valid answers systematically, and the three fixture defects were exactly that failure. A consistent grader can still consistently reject a correct answer.

For RBAC and NetworkPolicy, the Python checks were replayed on a real cluster: kind v0.33.0 with node image v1.34.3, using kindnet’s NetworkPolicy enforcement. The node dropped all new egress. Enforcement controls passed first: an open flow was allowed, deny-all blocked it, and RBAC grants and absences behaved. Then 30 inputs (originals, reference fixes, 13 GLM patches, and 5 failing non-GLM patches) were checked with kubectl auth can-i and real pod-to-pod probes. All 30 verdicts matched. That covers the six RBAC and NetworkPolicy cases only. Security group, IAM, CI, and Dockerfile checks are simulator-only.

The code tests also share a process with the model’s code inside the VM. A hostile patch could fake a pass. That’s acceptable for cooperative models, not for an adversarial benchmark.

Pins

All local rows ran on llama.cpp build b11429 (d81235049). Hosted rows pinned one provider with fallbacks disabled, required parameter support, and an FP8 filter, and every response’s served model and provider were recorded. Hosted catalog ids aren’t weight hashes, so I can’t prove the provider didn’t change what it serves. The rendered chat template for every local row was saved with the results, because identical messages don’t guarantee identical model inputs.

The harness, fixtures, and raw results aren’t public yet. If there’s interest, the fixtures and graders are the parts I’d publish first.