AI models can explore alien rules but can't do it reliably or apply what they find

Frontier language models can acquire genuinely unfamiliar rules through active experimentation — but they do so unreliably, and knowing a rule does not mean being able to use it. Those are the central findings of ExplorationBench, a benchmark built around two deterministic "alien" environments whose hidden semantics deliberately contradict both the supplied manual and anything a model might have memorized.

The two sandboxes are AlienCode, a small programming language with 31 hidden semantic rules (EMIT(100) prints 127 because integer literals are silently XOR-ed with 27), and AlienLogic, a natural-deduction proof system with 24 patched inference rules. Both are fully executable, so every answer is graded by an interpreter or proof-checker with no LLM judge involved. Before any exploration, no AlienCode trajectory exceeds 15.7% on held-out tasks. After four rounds of active probing, the best trajectory reaches 87.6%. The same number of model turns without environment feedback tops out at 11.0%.

Who designs the experiments matters enormously in AlienCode. Handing a system back its own best probes — without letting it choose them — lowers accuracy for 9 of 10 systems, by a median of 17.1 points, even though the evidence received is identical. Random grammar-valid probes barely help at all (median 5.7%). AlienLogic inverts this: there, what proofs are tried matters, not who chose them, and being told the complete rule set outperforms every system's own exploration (93–97% versus a best of 83.8%).

The dissociation between discovering a rule and applying it is striking. Tasks whose required rules a system states correctly in its report are still solved only 70.9% of the time. Two trajectories state both keystone positional rules correctly yet end at under 13%. The rule report is a lossy signal — useful diagnostically, but no substitute for held-out accuracy.

Reliability is the sharpest problem. Trajectories of the same system under identical budgets end up to 72.8 points apart in AlienCode, while answering noise (re-running the same question three times) contributes at most 4.7 points. Kimi K3 ends between 4.8% and 77.6% across its three runs; Gemini 3.8 Flash between 6.2% and 77.6%. Six of 30 AlienCode trajectories finish at least 3 points below an earlier milestone — exploration can actively undo progress. Rankings also shift depending on whether you score by best trajectory or mean: Qwen3.8-Max ranks sixth by Best@3 and third by Mean@3.

The benchmark's design choices are deliberate and defensible. Alien semantics prevent recall from substituting for exploration. Executable grading removes the need for expert verification. Separating probe selection, rule reporting, and held-out task performance into distinct metrics lets the benchmark distinguish finding a rule from using it — a distinction that turns out to matter a great deal.