Renaming symbols in SWE-bench repos drops agent scores by up to 7.4 points

SWE-bench scores may be lying to you — not because the benchmark is broken, but because the repositories it tests on are so familiar to frontier models that agents can navigate them partly from memory rather than reasoning.

SchrodingerRepo makes that case rigorously. The framework transforms each evaluation instance at runtime across four levels: rewriting the problem statement, remapping all internal namespaces to fresh invented names, reordering intra-file definitions, and rewriting issue-relevant code into behaviorally equivalent variants. The underlying task, execution environment, and test-defined correctness criteria stay identical. Only the surface representation changes.

The results are uncomfortable for anyone citing SWE-bench Verified numbers. Across four models — GPT-5.4-mini, GPT 5.1, DeepSeek-v4-Flash, and Gemini-3.1-Flash-Lite — full transformation drops Pass@1 by 6.0 to 14.4 percentage points. The degradation is statistically significant. More telling is what happens to interaction cost: input-token usage more than doubles in the worst-affected configurations, and 81.6–83.6% of the extra actions across both analyzed models go toward navigation, search, reading, and probing rather than editing or testing. Agents aren't failing because the bugs got harder; they're failing because they can no longer shortcut to the fix location using memorized naming conventions.

Namespace remapping is the single most damaging transformation. Renaming internal symbols — classes, functions, file paths — while leaving external APIs and language builtins untouched produces Pass@1 drops of 6.0 to 7.4 points on its own and more than doubles input-token consumption for some models. Problem statement rewriting, by contrast, barely moves the needle, which tells you the memorization is in the repository structure, not the issue text.

The temporally held-out SWE-rebench split provides a useful control. On instances created after the evaluated models' release dates, full transformation leaves Pass@1 unchanged at 17.27% for GPT-5.4-mini while still raising interaction cost. That asymmetry — cost up, success rate flat — confirms the framework isn't secretly making tasks harder; it's just removing a familiarity shortcut that wasn't available for genuinely novel repositories anyway.

A motivation experiment adds qualitative weight: human reviewers found that more than 65% of SWE-bench Verified instances show clear data-leakage evidence when models are probed with progressively revealed semantic units, and more than 18% can be recalled at the concrete patch level before the agent has seen any repository files.

The framework is currently Python-only and command-line-scoped, which the authors acknowledge as a limitation. IDE APIs and language-server tooling would require additional adaptation. But the core mechanism — treating repository representation as a seeded latent variable instantiated fresh per evaluation run — is the right architectural response to a benchmark ecosystem that has been training and testing on the same canonical codebases for years.