Skill 'evolution' has an answer key — the real problem starts where the key ends

Field report · Mike Arbuzov · SkillFed Research · August 06, 2026

AI-assisted notes · reviewed by SkillFed

Most self-improving-skill papers grade every candidate against an oracle, and their own ablations show the key is doing the work. Improving toward a form nobody knows has no key. There, grading collapses evolution into rules that only find what they already name.

Most papers about agents improving their own skills run the same machine. The agent drafts a skill, tries it on a task, and a grader that already knows what success looks like says pass or fail. Keep what passes, discard what fails, repeat. The grader takes different forms — a benchmark with reference answers, a hidden evaluation set (a quiz the student never sees, so it can't be memorized), a judge anchored to hand-labelled examples, a verifier that runs the code — but it is always the same object: an answer key. Something outside the loop that already knows what better means.

This works, and the papers are right that it works. A validation gate alone turns a 40% agent into an 89% one. Curated skills lift success 16.2 points while self-generated, ungraded ones cost 1.3. The results are real.

The word is what's wrong. This is not evolution. It is optimization against a trusted grader — hill-climbing with a GPS. And the papers' own ablations show it.

The answer key is doing the work

The cleanest experiments in this literature are the ones that remove their own grader and watch what happens:

take away	what happens
the validation gate	Skill writing lands at 40.1% — statistically the same as having no skills at all. The writing was never the hard part.
the executable check	Let agents add self-mined skills to a shared library ungraded and accuracy collapses from 60.3% to 36.3%, below several untrained baselines.
the isolated verifier	Remove the one component that never sees the generator's reasoning and a self-evolving system loses 30 points. The authors say why in one sentence: otherwise the reviewer rubber-stamps whatever the generator already believed.
the anchor set	Strip ten hand-labelled reference examples from an evolving evaluator and within one run, on every seed, it passes 94–100% of everything it grades.
the teacher	Replace external feedback with the agent's own and improvement flattens while alignment quietly declines — accuracy rises for a round, then collapses.

Read together, these say one thing: in the standard setup, the intelligence is in the answer key. The generation half is nearly interchangeable. Which is a perfectly good result about optimization — and a warning about what happens the day you need to improve something no key exists for.

The problem with no key

We index tens of thousands of community skills, and the artifacts we most need to improve are not the skills — they are the things around the skills: the conventions, the specifications, the formats, the instructions that tell a machine how to read all the rest. None of these has a best form written down anywhere. There is no benchmark for a good specification. There is no hidden eval set for a convention. Nobody holds the reference answer, because the reference answer does not exist yet — if it did, you would write it down and skip the loop entirely.

That is the situation real evolution operates in, and it is worth saying plainly how different it is. Evolution in the wild has no target form. Nothing knows what the best organism looks like; there is only better against the environment you actually meet — and the environment keeps changing, so the criterion moves too. Fitness is not a score against a key. It is friction against material you don't control.

An improvement loop in this regime has a property the answer-key papers never face: it is perfect at every round, by its own lights. The system defines what good means, then measures itself against its own definition, and passes. The stripped-anchor result above is this failure measured in a lab — take away the external reference and the evaluator approves everything — but there the fix was to put the key back. When no key exists, you don't get that fix. You have to run the loop anyway, knowing that every internal score is, in the limit, a tautology.

We have been running such a loop against our corpus for some time — the companion report covers its first fifteen cycles and what broke — and two failure modes showed up that we have not found described anywhere in the literature. Both are about grading. Neither is fixable by grading harder.

Failure one: grading collapses evolution into rules

To grade consistently you write checks. Checks are rules. And a rule can only detect what it already names.

That sounds harmless until you watch what it does to a closed loop. Selection driven only by named checks searches inside the named space — it finds endless instances of the defect categories it already has, and no new categories, ever. Across its first fifteen cycles our loop filed thousands of defect reports, every one inside an existing category. The number of genuinely new defect categories it produced was zero. Every new category the project has — three, each later confirmed on disjoint samples — was added by a person reading output, not by the graded pipeline.

The mechanism is more uncomfortable than "the checks were incomplete." The checks were improving. Of the eleven substantive edits made to our grader over that period, the loop itself made eight — and every one of the eight narrowed what the grader was allowed to report. Each was locally justified by real evidence: precision was up, noise was down, output became more countable. Not one added a new thing to look for. Each edit was right, and the sum was a grader that could no longer say anything it hadn't been shaped to say. The loop was right every single time and wrong overall — and it reported success the whole way down, because "satisfies the checks" was exactly what it was optimizing.

This is the answer-key machinery running in reverse. With an external key, tighter grading concentrates the search near the key, which is what you want. With a self-defined key, tighter grading concentrates the search near yesterday's understanding — and evolution stops precisely to the degree that the grading succeeds.

Failure two: the genuinely new arrives as chaos

Where did the three new defect categories come from, if not from the checks?

From prose. From the grader's free-form remarks — the sentences that ended "…but I have no category to file this under, so I am not filing it." Structured output is how you make grading countable: findings as records, categories from an enum, severities from a scale. It is also, we measured, a filter that silently discards exactly the observations evolution needs. When we gave our grader a free-prose channel alongside its structured one and then measured what the prose contained, just under two-thirds of 150 graded documents carried substantive observations the grader had declined to file — and every genuinely new defect category we have ever found appeared in that free prose first. The structured channel, across the loop's entire history, has never produced one.

The pattern generalizes past our loop, we suspect, because the cause is not a bad schema — it is what a schema is. Novelty arrives without a name. A schema is a list of names. So the moment your improvement loop consumes only schema-shaped signal, it is novelty-blind by construction, and it will optimize beautifully inside a frozen space while reporting green the entire time.

The consequence we'd defend as a general principle: a real evolution loop needs both substrates. Rule-based checks to count, compare, and select — selection has to be disciplined, and rules are the right instrument for it. And free-form reading as the medium where the unnamed can surface — because that is the only place it can. The chaos is not noise around the signal. For the part of the problem that matters most, the chaos is the signal, and the counting comes after.

The few papers near the real problem

A handful of the 191 papers in our research corpus stand at the edge of this regime, and they are worth naming. Self-play skill-writing with no labels and no verifier is the most honest attempt at key-less improvement — and its own headline finding is the collapse mode, with a fix that selects the best historical version rather than trusting the latest. The anchored-evaluator paper is the only one that evolves the grader itself — grading the grader — and it shows the whole safety load resting on a small fixed external reference. Three established optimizers handed back an untouched seed on 4 of 6 tasks: the local-optimum wall, hit by machinery rather than argument. And the plateau result — improvement flat by the tenth batch, drifting down by the thirteenth — is what a closed criterion looks like from outside.

Each of these maps one edge. None of them runs the full problem: a loop where the criterion itself is a moving target that must keep being re-derived from experience, because no fixed reference exists to hand it. In our corpus, no paper does. We think that absence is the most underrated fact about this research direction — the largest direction in the field is named after a process almost none of it performs.

What survives without a key

We are deliberately not describing our machinery here — what the free-form channel feeds, how selection is wired, what the loop edits. What we will state is the shape of what seems to survive contact with the problem, because each item was forced on us by a measured failure rather than chosen:

Score against material the loop doesn't control. The source documents, the environment, the input stream — external referents make a score falsifiable even when no key exists. Internal conformance is always, eventually, a tautology.
Keep the input unknown. A fixed benchmark optimized against repeatedly stops being external — the loop memorizes its way around it. Fresh, disjoint input every round is not a statistical nicety; it is what keeps "better" meaning anything.
Let the criterion move, on the record. When the best form is unknown, your definition of better should change as understanding improves. The discipline is to change it explicitly and restart the evidence, never to let it drift silently.
Feed selection from both substrates. Rules count; prose notices. A loop consuming only its own structured output has already frozen — it just hasn't noticed, and by construction it can't.
Suspect the instruments first. In a key-less loop the graders are part of the system being evolved, and all but two of the recorded failures in ours were in the measurement, not the artifact. An instrument that is wrong is not degraded — it points somewhere, confidently, and the somewhere is wrong.

Scope, stated plainly

The literature claims are corpus-relative. The SkillFed research corpus holds 191 papers at a 2026-07-21 cutoff, read with model assistance. "Most papers use an answer key" is argued from the ablation evidence above, not from a paper-by-paper census, and the corpus skews to 2026 preprints that are largely unreplicated.
The two failure modes are measured in one loop. One corpus, one domain, one operator. The numbers behind them — the zero, the eight-of-eleven, the two-thirds — are ours, and the claim that the modes generalize is an argument, not a measurement.
The design of our own loop is deliberately withheld. This piece argues that the problem is misunderstood and harder than the benchmark results suggest; it does not claim we have solved it.

Related: fifteen cycles of a self-improving spec — where the failures actually were, and the census of the corpus this loop runs against. The research directory shelves all 191 notes, including the evolution direction this piece is about.