Past experience predicts model confidence better than re-reading current reasoning

Every existing confidence estimator reads the same thing: the inference that just happened. Verbalized confidence asks the model to introspect on its own answer. Likelihood methods score token probabilities. Consistency methods resample the task and count agreement. All of them are, in the paper's framing, trace-intrinsic — they look at the current derivation and nothing else.

XConf breaks from that premise entirely. The core claim is that past graded experience is a better basis for confidence than any re-reading of the present inference. The mechanism has two stages. Recall retrieves episodes from a stored bank of the model's own prior attempts — keyed by task similarity and stated confidence — and reads off the historical hit rate: how often did this model, feeling this confident, on problems like this, actually get it right? Reflect then shows those retrieved episodes back to the model as short cards, asks it to name any recurring failure mode, and has it restate a calibrated confidence. The final estimate averages the two readings.

The bank itself is a by-product of normal operation: tasks, reflections, stated confidences, outcomes, and a one-time lesson written after grading arrives. No weights are updated. No logit access is needed. One answer generation is the cost.

Across nine benchmarks and four models from three families, the method beats or matches ten-sample self-consistency in discrimination on 23 of 24 comparisons, with substantially lower calibration error, at a tenth of the generation cost. The advantage is largest exactly where sampling-based methods struggle: code, where exact-match voting is degenerate, and agent tasks, where resampling a thirty-step rollout ten times is impractical.

Several findings sharpen the picture. Calibration keeps improving as the bank grows, with no saturation on the domains where episodes are scarcest. A bank borrowed from a different dataset of the same kind transfers well; a bank borrowed from a different model loses ground specifically on tasks where the two models' errors diverge — a clean analogue of the human finding that people predict their own accuracy far better than someone else's. Self-generated outcome labels fail: the model labels its confidently-wrong episodes as successes, and the bank then propagates that blindness forward. An independent but imperfect judge, agreeing with ground truth only some of the time, preserves most of the value.

The one honest loss is on short votable factual recall, where self-consistency remains the stronger discriminator. The paper is direct about this: the claim is breadth and cost where voting cannot go, not superiority everywhere.

For anyone building systems that need to decide what to ship, escalate, or retry — which is most production agent work — the operational implication is concrete: abstaining on the least-confident episodes raises delivered success rates measurably on agent tasks, and the estimate that drives that decision costs one generation plus one short retrieval call.