We built a loop to improve a spec — sixteen of its worst failures were in the measurement

Field report · Mike Arbuzov · SkillFed Research · August 05, 2026

AI-assisted notes · reviewed by SkillFed

Fifteen cycles, 2,050 compilations over 750 skills and a 53-entry change ledger later, the specification improved and the instruments kept lying: of sixteen recorded failures only two were coding bugs, and the tool written to stop us confirming results early nearly confirmed one.

Skills are written for people to read. We compile them into a form written for machines to read — one where every fact in a skill has a stable address, so two versions of the same skill can be diffed, a family of near-duplicates can be merged, and an update can touch one claim instead of rewriting a document. The compiler is a specification. Like any specification, it is wrong in ways nobody can list in advance.

The usual way to improve a spec is to read the output, form an opinion, and edit. That does not scale past a few dozen documents and it has no memory: the reason for each edit lives in whoever made it. We wanted the spec to improve against the corpus instead — measurably, with a record, and with the discipline that a change which does not work gets reverted rather than defended.

Fifteen cycles later we have 2,050 encodings over 750 skills, 3,973 defect claims, and a change ledger of 53 entries covering every edit, its evidence, and its verdict. The compiler improved. That is not the interesting part.

The design decision everything rests on

A reviewer reads each compiled document against its source and reports defects. The obvious design is to let it propose fixes to the spec. We do the opposite, and it is the one decision the whole thing depends on: the reviewer localizes and never proposes.

Every defect is addressed to the spec rule that should have prevented it:

{ "check": "OMISSION",
  "source_quote": "Requires plan tier Pro or above",
  "output_quote": "ACTION: call the endpoint",
  "class": "COMPILER",
  "locus": "X.REQUIRE.2",
  "needed": "encode the plan-tier requirement as its own precondition line" }

The reason is arithmetic. Fifty different proposals for fixing the same rule cannot be added together; they can only be argued over. Fifty defects addressed to that rule can: group by rule, count distinct documents, rank. Localization is what turns a pile of criticism into a gradient with coordinates, and it is why the spec is written as numbered, individually addressable rules rather than prose. Without addressable rules there is nothing for the evidence to point at.

Two filters run before anything is counted. A finding is evidence for a rule only while that rule's text is unchanged since the finding was made — editing a rule invalidates its own history, which is correct and occasionally expensive. And 1,287 of 2,675 omission claims were mechanically refutable: the content was in the output the whole time. We keep the refuted ones, because the refutation rate is how we measure the reviewer.

What editing a spec actually taught us

One edit per cycle, under a token budget. Then the step that matters: re-run the same documents through the edited spec and check that the defect it was written to fix actually fell. A text edit, unlike a real gradient, guarantees nothing — it can point the wrong way, and twice it did.

The clearest result is what shape an edit has to take. The same intent, in three drafts:

draft	shape	outcome
"Preserve requirements faithfully."	prose exhortation	0 for 3 — nothing to check, nothing changed
"A stated plan tier produces its own precondition line."	per-line, countable	4 for 4 — the shape that works
"A requirement governing several steps heads them as one block."	a relationship between lines	reverted

The third one is the finding we did not expect. Every successful edit had specified the form of a single line. The moment we specified a relationship between lines, the compiler produced more of the surface form and none of the structure — it emitted more guards, each still governing exactly one step. It executes shape, not scope.

A deletion taught us the mirror of this. One rule instructed the compiler to consult an input that has never existed in roughly 700 runs, and the mechanism it served produced a vocabulary that was 90% single-use. Removing it made the output measurably worse. The dead clause was doing real work: because its condition could never be satisfied, it licensed the behaviour unconditionally. A rule can be dead by reference and live by effect — which retired our heuristic that a rule nothing ever complains about is a rule earning nothing.

Every edit carries its own evidence and its own verdict, including the ones that failed:

{ "cycle": 13, "locus": "X.REQUIRE.2", "op": "INSERT",
  "evidence": { "rate": 0.20, "batches": 9, "documents": 90, "noise": 0.04 },
  "outcome": "REVERTED",
  "result_note": "1 fixed, 1 partial, 1 unchanged, 1 worse — does not decrease the objective it was derived from" }

Three edits were reverted outright and a fourth rule was removed when its confirmation window closed against it. We think that rate is too low rather than too high.

Every hard failure was in the instrument

Twenty failures are on the register. Two were coding bugs. The rest were statistical judgement, and every one of them produced a plausible number that we believed for a while.


Compression reported at 55%	true value 5.1% — the API's input count includes the spec itself
A conformance check flagged 462 violations	459 were correct usage
"Mutations tripled"	one document supplied 555 of 788
An edit confirmed on two same-direction reads	contradicted by the third; two falls happen ~25% of the time by chance
A step in the approved plan	went nine cycles unrun — then the resulting noise was diagnosed as a limit of the design
The tool built to enforce the confirmation rule	reported CONFIRMED on its first run, by pooling two different reviewer versions into one baseline

That last one is the whole report in miniature. The instrument written specifically to stop us confirming things too early nearly confirmed something too early, because it mixed two measuring devices and inflated the baseline's spread from 6 points to 22.

None of these were detectable by looking at the number. They were detectable by looking at the shape of the number — which document contributed it, which instrument produced it, whether the median agreed with the total. That check now runs on every rate we report, and it has reversed the conclusion four times in the last three cycles.

The compiler and the spec were the easy part.

Where it stopped, and why we are saying so

Cycle 15 recorded a local optimum. Six consecutive decisions read: weak-keep at a third of a standard deviation, keep-and-watch later reverted, revert, revert, and two holds. The four rules at the top of the evidence table have absorbed six edits between them and return at the same rates. What is left there is semantic damage — conditions weakened, lists summarised, qualifiers dropped — that the compiler has not fixed from instruction in six attempts.

Two of those cycles closed off an approach on evidence rather than on taste, which is the part we would keep. The loop's own rules now carry a change trail for the same reason: an audit found the layer enforcing "one edit, one test, one record" on the spec had itself changed ten times with no record at all, and had once been cited as authority for a rule that did not exist.

Method and limits

Batches of fifty documents, drawn disjointly from one corpus, compiled by a small model at temperature zero and reviewed by a larger one. One domain, one corpus, one operator.

Fifty is too few to hold out a validation set: the defects that matter occur at 4–20%, so a split large enough to see them is most of the batch. We spend the sample on the same-batch re-run instead, where the measurement carries no sampling noise, and accept that this is training-set performance and proves less.

The important limit is that we have never tested whether any of it helps. Nothing here measures an agent using a compiled skill against one reading the original — and one paper in our own research index found a re-encoded text scoring below no text at all. Every number in this report is a proxy, measured against a representation whose value to a reader is undemonstrated. That experiment is one batch of work, it is designed and pre-registered, and it outranks another edit to the spec. It is the next thing we run.

We claim no outcome for the compiled representation, in search quality or anywhere else. What we claim is narrower: a specification can be improved against a corpus with a record and a rejection path, the failure modes are in the measurement rather than the spec, and the loop will tell you when it has stopped working if you build it to be able to say so.