AnyJev makes LLM logits trustworthy enough to actually automate decisions

Raw logits from an LLM are not probabilities. Ask a 20-way classification question and flip the option order: on Qwen3-8B with BANKING77, the answer changes nearly a quarter of the time. The confidence readout is so poorly calibrated that only 7.7% of decisions can be safely automated at a 5% error budget — meaning almost everything gets routed to a human anyway.

AnyJev attacks both problems without touching the model's weights. The zero-label fix (L0) is conceptually clean: instead of reading logits once, it reads them over every cyclic rotation of the options, averages out the position bias, and divides out an estimated label prior. Order-flip rate drops from 0.230 to 0.073 on that same benchmark. Auto-decidable traffic jumps from 7.7% to 46.3%. Accuracy moves from 0.747 to 0.803 — real, but not the point. The point is that a calibrated probability lets you set a threshold and act on it.

With 100 to 500 labels, L1 adds temperature scaling on top of L0, cutting calibration error from 0.240 to 0.095. L2, the newest level, goes further: it fits a closed-form linear head (shrunk LDA or ridge) on the hidden state at roughly two-thirds of the model's depth. One truncated forward pass, no gradients, solved in seconds on a CPU. On the typed-decisions benchmark — 20 questions, 300 labels each — a Qwen3-1.7B head at L2 reaches 0.730 accuracy, matching what the original Jev paper reports for its fine-tuned model. A Qwen3-4B head at 0.786 ties a fine-tuned 421M Laya. The L2 forward costs 0.68× of a plain full forward on Qwen3-8B, because the model stops at block 24 of 36.

The self-maintaining angle is worth taking seriously. Once a head is fit, it tracks question rewordings without new labels: 30 unlabelled requests of a reworded prompt bring accuracy back from a drop to 0.65–0.70 up to 0.74–0.75, against 0.77 for a full refit. New option sets need labels; new base models need a re-solve from stored states; wording drift handles itself.

The limitations section is unusually candid. The typed-decisions gold labels are teacher-LLM agreement, and that teacher agrees with itself only 0.735 of the time — so the accuracy ceiling is not 1.0. Coverage at 5% risk is a high-variance estimate at n=300. L0 can hurt accuracy when one label dominates the prior. L2 needs hidden states, which means the transformers backend today and vLLM/SGLang on the roadmap. The whole evaluation is isolated decisions, not decisions inside a running agent loop — which is the next stated milestone.

For anyone building routing, triage, or guard logic on top of open LLMs, the core offer is concrete: typed questions with probabilities you can threshold, served from a single prefill, with a clear upgrade path from zero labels to a few hundred.