llm-as-a-verifier/llm-as-a-verifier

The core insight here is that LLM-as-a-Judge collapses a probability distribution into a single discrete score and throws away everything else. LLM-as-a-Verifier keeps the full logprob distribution over score tokens, takes an expectation across it, and then averages that expectation over multiple criteria and repeated evaluations. The result is a continuous reward signal in [0, 1] rather than a coarse integer label.

That signal does real work. On Terminal-Bench 2.1, selecting the best of five trajectories with the same model that generated them — self-verification, which should be the hard case — lands at 88.0% (±0.6%), well above the Pass@1 baseline of 78.7% and closing meaningfully toward the oracle ceiling of 96.6%. SWE-Bench Verified and MedAgentBench show similar gaps between the base Pass@1 and what selection recovers.

The selection mechanism is a Probabilistic Pivot Tournament. A naive round-robin over N candidates costs O(N²) pairwise comparisons. PPT cuts that to O(Nk) by first running a random Hamiltonian cycle to seed rough rankings, then comparing every candidate only against a small pivot set drawn from the top of that initial ranking. Positional bias is cancelled by alternating which candidate appears in the A versus B slot across repeated evaluations. The pivots parameter directly trades verification cost against ranking accuracy.

Progress tracking is the less obvious application. The track function scores a completed trajectory at each checkpoint step, producing a curve that rises as the agent makes genuine progress and stays flat or drops when it goes wrong. The online ProgressTracker variant does the same thing incrementally — each call to update returns a live score so a harness can abandon a failing rollout before it wastes its full budget. The prompt template explicitly instructs the verifier to trust observed outputs rather than the agent's own narration of what it did, which is a small but pointed design choice.

Prefix caching is handled deliberately: the criterion text is placed at the tail of the prompt so the much longer task description and both trajectories form a shared prefix that can be reused across criteria and repeat evaluations. The README reports this takes the cache hit rate from around 5% to around 78% on Terminal-Bench 2.1, cutting uncached input tokens by roughly 3.4×. Token accounting is built into the library and measured from actual backend usage blocks, not estimated.

The framework accepts images at every entry point, routing them to multimodal backends like Gemini 2.5 Flash or a locally served vision model. Per-step frames accumulate across ProgressTracker updates so the verifier always sees the full visual history — relevant for robotics rollouts where the camera feed is the ground truth.

The TurboAgent proxy wraps all of this as a drop-in API layer for Claude Code, generating parallel candidates and selecting among them transparently. That is either a convenience or a footgun depending on whether you want visibility into what the verifier is actually doing.