A cheap decision-only judge plus a confidence gate gets you 99% of GPT-6 accuracy

The central finding here is architectural: a decision-only judge that returns typed label probabilities rather than generated text can match a frontier LLM judge on most practical evaluation tasks, and its confidence score reliably identifies the cases where it cannot.

The judge under study, TypeSafe JEV, takes natural-language instructions and structured inputs and returns a verdict plus a probability distribution over allowed labels. No rationale, no chain of thought. On ordinary preference ranking and evidence-grounded factuality it lands within three percentage points of GPT-6, the strongest comparator tested, while costing roughly two orders of magnitude less per thousand judgments. On the harder JudgeBench suite, which stresses derivation-checking and multi-step reasoning, the gap widens to about fifteen points - and human adjudication confirms that gap is real, not label noise.

What makes the paper practically useful is the confidence cascade. JEV's maximum label probability tracks its error rate monotonically: items where the model assigns very high confidence are nearly as accurate as GPT-6 on those same items, while items where it is uncertain are exactly where GPT-6 pulls ahead. A frozen routing policy that accepts confident verdicts and escalates the rest to GPT-6 retains 99% of GPT-6's accuracy at roughly half its fee. That is not a theoretical bound - the paper tests pre-specified thresholds on held-out data.

The limits are stated with unusual precision. Style-adversarial pairs, where the wrong answer is more elaborately written, expose a specific weakness: JEV's accuracy drops nearly ten points on those pairs while GPT-6 barely moves, and the confidence signal degrades there too, with error-detection AUROC falling noticeably on hard style pairs. Reference-free prose judgment is a boundary for every judge tested, not just JEV - all three models evaluated on balanced general-response hallucination labels perform near chance while remaining highly confident, a combination the paper flags as a reason to avoid any automated judge as the sole arbiter of consequential decisions.

The calibration analysis adds nuance. Temperature scaling fitted on a pilot selection set transfers unevenly across tasks, with the optimal temperature pointing in opposite directions for different benchmarks. No single scalar fixes calibration globally; each deployment needs local validation. The paper is explicit that a usable verdict, a correct decision, and reliable uncertainty are three separate properties, each requiring independent verification.

The experimental design is careful about what it can and cannot claim. The multi-family model comparison was added after initial results were known and is labeled exploratory throughout. Human adjudication of judge disagreements used one team-member annotator with an LLM second pass rather than independent naive raters - a deviation the paper documents in an amendment log. Benchmark contamination and training overlap are unknown for all models.

For anyone running evaluation pipelines at scale, the operational numbers are the point: a decision-only first stage with a confidence gate can absorb the majority of judgments cheaply, reserving expensive reasoning models for the fraction where they actually change the answer.