Jev in 25 lines: building the decision model locally

TL;DR

In the design review I concluded that Jev, TypeSafe AI’s typed decision model, is cheaper and faster than an LLM and not more accurate. That raised an obvious build-vs-buy question: if the value is “cheap probability over a fixed set of options,” can I get that from a small open model running on hardware I already own?

Mostly yes, with one catch that matters. The technique is to prompt with lettered options, run one forward pass, and read the logits for just the option letters. A parody post called “Jev in 25 lines of Python” does exactly this, and the technique under the joke is real. The catches are calibration (raw softmax is bad until you fit it) and serving (Ollama could not do it when I wrote the design).

Same disclaimer as the review: nothing here is built or measured by me. This is a plan and a sketch, with other people’s numbers labelled as theirs.

The trick

A Jev Choice question gives you labelled options, and you get back a probability per option. You can get that shape from any causal language model:

1. Put the options in the prompt as lettered lines (A, B, C…).
2. End the prompt where the answer letter would go.
3. Run one forward pass. Do not generate.
4. Take the logits at the final position, for only the token ids of the option letters.
5. Softmax over just those.

You never generate text, so there is no rationale and no parsing, and the output is typed by construction. It is a standard constrained single-token classification, the way multiple-choice benchmarks have been scored for years. The “25 lines” post, which disclaims being a serious alternative, uses a sub-1B open-weight model through llama-cpp-python with logits_all=True. These five lines are the core, as published there:

logits        = model.scores[model.n_tokens - 1]
token_ids     = [model.tokenize(text=label.encode(), add_bos=False)[0] for label in labels]
choice_logits = numpy.asarray([logits[token_id] for token_id in token_ids])
logprobs      = choice_logits - numpy.logaddexp.reduce(choice_logits)
probabilities = numpy.exp(logprobs)

Around it, my wrapper would look roughly like this. This is a sketch I have not run. The five lines above are the upstream technique; the rest is my outline of how I would frame it.

import numpy
from llama_cpp import Llama

model = Llama(model_path="small-instruct.gguf", logits_all=True, n_ctx=4096)

def choose(state: str, question: str, options: dict[str, str]) -> dict[str, float]:
    labels = list(options)  # e.g. ["A", "B", "C"]
    lines = "\n".join(f"{k}. {v}" for k, v in options.items())
    prompt = f"{state}\n\n{question}\n{lines}\nAnswer:"
    model.reset()
    model.eval(model.tokenize(prompt.encode()))
    # ...the upstream lines above go here...
    return dict(zip(labels, probabilities.tolist()))

Two things I would check before trusting it. Whether the letter’s first token is the one the model actually emits after Answer: (a leading space versus none changes the token id), and that every option letter maps to a single distinct token.

Why bother, given Jev exists

Here is the comparison from my design doc, with the local side labelled as what other people reported:

The line that changes my mind is the calibration one. Raw softmax over choice logits is badly calibrated, so a naive local version would be worse than Jev on exactly the dimension that matters when you threshold on confidence. But with temperature fitting, the reported figure goes from 0.466 to 0.081, which is below Jev’s 0.107. Calibration is not the vendor’s moat, as long as I do the fitting. And the design already commits me to fitting thresholds on my own labelled data regardless of backend. So the work is the same either way. Only who hosts the weights changes.

The other row worth highlighting is images. Jev cannot see them. A local vision-language model read at the logits can. I am not building that now, but it is the one capability a hosted Jev cannot match.

The ecosystem already exists

I would not hand-roll this when people are doing it carefully. These come from an alternatives roundup and the projects’ own pages:

• mini-jev: a frozen 4B open-weight instruct model, Choice and Noul, about 8.5 GB, Apple Silicon or CUDA. It explicitly disclaims calibration: a ranking with a confidence gap, not calibrated probabilities. It serves a Jev-compatible endpoint shape, so swapping it in should be close to free.
• logit-classifier: Choice, Score, and Noul, with a label prior and temperature fitted on a public intent dataset. About 140 ms per record on a 3090 Ti, needing 10 GB or more of VRAM. The vision-language variant reads images.
• SemIf: a 4B model for typed decisions, no calibration claimed.
• Kev and Laya: Kev publishes training code and eval sets benchmarked against Jev (Brier 0.237). Laya is where the 0.466 to 0.081 figure comes from.

Why Ollama can’t serve it

My Mac Studio already runs Ollama, so that was the first place I looked. It does not work, and the reason is specific.

The trick needs the log-probability of one particular token, the option letter. Ollama reports logprobs only for the top K. If the letter you care about is not in the top K, it comes back absent, and absent reads as probability zero. That is the same silent-wrong-answer failure the whole design is built to avoid: the model’s second-favourite option vanishes and the arithmetic still produces a clean-looking distribution.

The upstream issue is ollama/ollama#18579, “Cannot get the log probability of a specific token unless it ranks in top_logprobs.” When I wrote the design in late September I recorded the proposed fix, PR 18580, as open. I rechecked while writing this post: the issue is now closed, and that PR was closed without merging on 2026-09-22. The author dropped it in favour of a smaller change, raising the top_logprobs cap, and I have not verified whether that shipped. If the cap is high enough to cover a handful of option letters, Ollama might work after all. Until I confirm that with a test against the installed version, my assumption is that it does not. It would need llama-cpp-python, vLLM, or one of the projects above running as its own service.

The plan: shadow first, local second

I do not want to build a model server only to learn my questions were wrong. So the order is:

1. Hosted Jev for the shadow phase. No ops cost, cents per run, and the thing being tested at this stage is whether my questions and aggregation work at all.
2. Once a question set has earned its keep, run the same questions on the same labelled split through a local backend and compare accuracy, calibration after fitting, and latency on my own data.
3. Whichever wins on my data, wins. If local is within noise, default to local: no cost, no egress, no vendor risk.

The wrapper that talks to the backend is thin and swappable by design, so step 2 changes one component, not the questions, the decomposition, or the aggregation.

How I would validate it

Same protocol as the hosted version, because it is backend-independent:

• Take 1,000 to 2,000 historical rows where a human’s final decision is the label.
• Split in half. Fit the temperature and the per-question thresholds on one half, and score the other.
• Report accuracy and calibration error on the held-out half, and look at the confidence band where accuracy actually meets the bar, not the vendor’s or project’s headline number.
• One threshold per question and per action, scaled to the cost of being wrong.
• Plant instruction strings in text fields and confirm the probabilities do not move.
• Pin the model file hash and log it on every row. Changing the weights means redoing the evaluation.

The result of the bake-off will probably be boring, which is the good outcome: a local backend that is within noise and costs nothing. I will write up how I think about when a classifier like this earns its keep, and how I build a labelled set to measure against, in follow-up posts.

What I would watch for

• Position and letter bias. Small models favour “A.” Shuffle option order across runs and look at the spread.
• Domain shift. Calibration fitted on one distribution degrades on another. The reported 0.081 is on a public intent dataset, not on my data.
• No abstain. The same hole as hosted Jev. Every question gets an explicit “insufficient evidence” option, and I check how often it wins.
• Still advisory. Whatever backend, its output sits in its own table behind a kill switch, and code I own aggregates the signals. See the review for why.