Even GPT-6 Astra drops from 100% to 64% when coding tasks require compositional depth

Most coding agent benchmarks hand the agent a GitHub issue and ask it to fix something. ProgramDistill flips that assumption: the specification doesn't exist yet, and the agent has to discover it by interacting with a working reference application whose source is hidden. Only after inferring what the software does must the agent reproduce that behavior in an incomplete codebase.

The benchmark is built through a three-stage automated process — mine, craft, patch — that runs over 26 web applications without human-written issues or annotations. Mining explores each live app to collect replayable browser-interaction traces, clean-re-collects them for conciseness, and admits only those that survive deterministic replay verification. Crafting removes the source implementations behind verified behaviors and checks that failures are confined to the intended targets. Patching is the evaluation stage itself, where coding agents must restore the missing functionality by observing the reference. Across 26 apps, this yields 1,975 replay-verified behaviors and 4,063 tasks.

The key structural idea is restoration depth: behaviors are organized into prerequisite lineages, so tasks can require recovering one behavior or up to eight dependent ones in sequence. At depth 1, GPT-6 Astra solves every task — 100% binary success. By depth 8, that falls to 64.0%. Claude Opus 5 drops from 96% to 32% over the same range. The decline isn't explained by longer context windows alone; within matched context-length bands, performance still falls sharply with depth, suggesting the challenge is genuinely compositional.

Trajectory analysis reveals something instructive about how agents fail. As restoration depth grows, the mean number of code lines to restore increases roughly ninefold, but reference observation steps per repair target fall by about 75%. Agents allocate less effort to understanding each required behavior precisely when they need to understand more of them. Astra, the strongest performer, inverts this pattern — it makes the most observation steps and the fewest edit/write steps among all nine models, averaging 96.3 current-app observation steps per trajectory against only 9.9 edits.

Full-application reconstruction, starting from a minimal scaffold rather than a masked codebase, is harder still. Astra reaches 49.2% cumulative workflow recovery; Opus 5 reaches 28.8%. Failure analysis across 977 missed behaviors finds that nearly 60% were never observed in the reference at all — agents simply didn't explore enough of the running application. Of the failures where the behavior was observed, most involve wrong state, route, or result rather than complete omission. Agents validated their implementations, but their checks were often too narrow: syntax passing, a successful API request, or a text match that didn't exercise the exact workflow that would have exposed the bug.

The benchmark is not yet publicly released, which limits immediate reproducibility. But the design — replayable traces as both task units and verifiers, prerequisite lineages as a curriculum axis — is a coherent answer to a real gap: evaluating agents on the kind of reference-guided development that happens constantly in practice.