llm-as-a-verifier/llm-as-a-verifier

The core complaint against LLM-as-a-Judge has always been that collapsing a model's uncertainty into a single discrete score throws away information. This framework takes that complaint seriously. Instead of asking a verifier model to output a number and reading it literally, it reads the full log-probability distribution over score tokens and computes an expectation across multiple criteria and repeated evaluations. The result is a continuous reward in [0, 1] rather than a coarse integer, and that continuity is what makes downstream uses—best-of-N selection, step-by-step progress tracking, early rollout termination—actually tractable.

The selection mechanism is worth understanding on its own. A naive best-of-N approach requires O(N²) pairwise comparisons. The Probabilistic Pivot Tournament here cuts that to O(Nk) by running a single Hamiltonian ring pass first, using those scores to identify a small pivot set, then comparing every candidate only against those pivots. Positional bias in pairwise prompts is handled by alternating which trajectory appears in the A slot versus the B slot across repeated evaluations—a small but meaningful detail that most judge frameworks ignore entirely.

The progress-tracking mode is the less obvious application and arguably the more interesting one. ProgressTracker scores a trajectory at each step as it arrives, without access to future steps, which means it can flag a hopeless rollout while it is still running rather than after the fact. The example in the README shows a successful Terminal-Bench trajectory with monotonically increasing scores and a failed one with consistently low scores—the signal is legible even mid-run.

On the benchmarks the README reports, the verifier consistently closes roughly half the gap between Pass@1 and Oracle. On SWE-Bench Verified, best-of-3 selection moves from 76.1% to 78.2% against an oracle ceiling of 84.4%. On MedAgentBench best-of-5, it moves from 70.2% to 73.3% against a 75.0% oracle. The self-verification result on Terminal-Bench 2.1—where the same model that generated the trajectories also verifies them—is the most striking: best-of-5 reaches 88.0% against a 96.6% oracle, well above the 78.7% Pass@1 baseline.

The prefix-cache optimization is a practical engineering detail that matters at scale. Verification prompts on Terminal-Bench 2.1 run to roughly 80k tokens each because they carry two full trajectories. By placing the criterion at the tail of the prompt, everything before it becomes a shared prefix that the backend can cache. The README reports this lifts the cache hit rate from 5.2% to 78.4%, cutting uncached input tokens by about 3.4×. Token accounting is built in and measured from the backend's own usage blocks, not estimated.

Multimodal inputs are supported wherever the verifier backend supports them—images can be passed per-call or per-step, and per-step frames accumulate so the verifier always sees the full visual history. The TurboAgent proxy wraps the whole thing as a drop-in replacement for the Anthropic API, so Claude Code can use it without any changes to the client.