QuoteBench: How Matched Scores Can Hide Command-Path Failures

A matched execution score on a coding-agent benchmark can be the sum of two large numbers that nearly cancel. QuoteBench makes that cancellation visible by fixing the model's reply and changing only the execution path it travels through.

The mechanism is concrete: many deployed agent systems pass a model-generated Bash command through an intermediate layer—an SSH remote, a Docker exec, a CI step—that wraps the command inside double quotes before handing it to a shell. That second parse eats the quoting the model already did. QuoteBench adds exactly one such parser as a controlled intervention, then replays stored replies through both paths without making new model calls.

The damage is large and consistent. Across eight same-window configurations, replaying the same reply through the added parser drops success by 55.4 to 73.2 percentage points. Crucially, the loss is not confined to adversarial payloads: the 14 benign control tasks alone lose 28.6 to 57.1 points, because models emit expansion-active characters even for ordinary commands. Exit codes miss a substantial share of these failures—between roughly a quarter and nearly half of failing executions exit zero while leaving the wrong final state, so any benchmark that trusts return codes silently misses them.

The benchmark then crosses the execution transport with the generation contract. When models are told their reply will be interpolated inside double quotes, six of eight configurations recover 30.4 to 60.7 points on the nested path—but pay for it on the raw path, because they rewrote their commands for the declared boundary. That trade-off is the point: the matched score accurately describes its declared path, but it hides whether the command survives a different one. GPT-5.6-sol's matched gap is near zero even though fixed-reply transport costs 64.3 points and contract-conditioned compensation restores 60.7. Two large opposing effects produce a small difference that looks like stability.

The deployment configuration also reorders models. The one reversal that is unambiguous at this resolution—GPT-5.6-sol versus Gemini-3.5-Flash—flips direction entirely between the raw and nested paths. Four more reversals sit on single-task margins.

The fixes are not novel: correct escaping at the interpolation point or executing the reply as a temporary script each restore every raw-path success in the replays. The contribution is the measurement, not the repair. The paper's prescription follows directly—report the generation contract, the execution transport, the operating point, and a final-state validator, because a matched score is a property of a specific path, not of the model.