We built a loop to improve a spec — sixteen of its worst failures were in the measurement

Field report · Mike Arbuzov · SkillFed Research · August 05, 2026

AI-assisted notes · reviewed by SkillFed

Fifteen cycles, 2,050 compilations over 750 skills and a 53-entry change ledger later, the specification improved and the instruments kept lying: of sixteen recorded failures only two were coding bugs, and the tool written to stop us confirming results early nearly confirmed one.

Skills are written for people to read. We compile them into a form written for machines to read — one where every fact in a skill has a stable address, so two versions of the same skill can be diffed, a family of near-duplicates can be merged, and an update can touch one claim instead of rewriting a document. The compiler is a specification. Like any specification, it is wrong in ways nobody can list in advance.

The usual way to improve a spec is to read the output, form an opinion, and edit. That does not scale past a few dozen documents and it has no memory: the reason for each edit lives in whoever made it. We wanted the spec to improve against the corpus instead — measurably, with a record, and with the discipline that a change which does not work gets reverted rather than defended.

Fifteen cycles later we have 2,050 encodings over 750 skills, 3,973 defect claims, and a change ledger of 53 entries covering every edit, its evidence, and its verdict. The compiler improved. That is not the interesting part.

The design decision everything rests on

A reviewer reads each compiled document against its source and reports defects. The obvious design is to let it propose fixes to the spec. We do the opposite, and it is the one decision the whole thing depends on: the reviewer localizes and never proposes.

Every defect is addressed to the spec rule that should have prevented it:

{ "check": "OMISSION",
  "source_quote": "Requires plan tier Pro or above",
  "output_quote": "ACTION: call the endpoint",
  "class": "COMPILER",
  "locus": "X.REQUIRE.2",
  "needed": "encode the plan-tier requirement as its own precondition line" }

The reason is arithmetic. Fifty different proposals for fixing the same rule cannot be added together; they can only be argued over. Fifty defects addressed to that rule can: group by rule, count distinct documents, rank. Localization is what turns a pile of criticism into a gradient with coordinates, and it is why the spec is written as numbered, individually addressable rules rather than prose. Without addressable rules there is nothing for the evidence to point at.

Two filters run before anything is counted. A finding is evidence for a rule only while that rule's text is unchanged since the finding was made — editing a rule invalidates its own history, which is correct and occasionally expensive. And 1,287 of 2,675 omission claims were mechanically refutable: the content was in the output the whole time. We keep the refuted ones, because the refutation rate is how we measure the reviewer.

What editing a spec actually taught us

One edit per cycle, under a token budget. Then the step that matters: re-run the same documents through the edited spec and check that the defect it was written to fix actually fell. A text edit, unlike a real gradient, guarantees nothing — it can point the wrong way, and twice it did.

The clearest result is what shape an edit has to take. The same intent, in three drafts:

draft	shape	outcome
"Preserve requirements faithfully."	prose exhortation	0 for 3 — nothing to check, nothing changed
"A stated plan tier produces its own precondition line."	per-line, countable	4 for 4 — the shape that works
"A requirement governing several steps heads them as one block."	a relationship between lines	reverted

The third one is the finding we did not expect. Every successful edit had specified the form of a single line. The moment we specified a relationship between lines, the compiler produced more of the surface form and none of the structure — it emitted more guards, each still governing exactly one step. It executes shape, not scope.

A deletion taught us the mirror of this. One rule instructed the compiler to consult an input that has never existed in roughly 700 runs, and the mechanism it served produced a vocabulary that was 90% single-use. Removing it made the output measurably worse. The dead clause was doing real work: because its condition could never be satisfied, it licensed the behaviour unconditionally. A rule can be dead by reference and live by effect — which retired our heuristic that a rule nothing ever complains about is a rule earning nothing.

Every edit carries its own evidence and its own verdict, including the ones that failed:

{ "cycle": 13, "locus": "X.REQUIRE.2", "op": "INSERT",
  "evidence": { "rate": 0.20, "batches": 9, "documents": 90, "noise": 0.04 },
  "outcome": "REVERTED",
  "result_note": "1 fixed, 1 partial, 1 unchanged, 1 worse — does not decrease the objective it was derived from" }

Three edits were reverted outright and a fourth rule was removed when its confirmation window closed against it. We think that rate is too low rather than too high.

Every hard failure was in the instrument

Twenty failures are on the register. Two were coding bugs. The rest were statistical judgement, and every one of them produced a plausible number that we believed for a while.


Compression reported at 55%	true value 5.1% — the API's input count includes the spec itself
A conformance check flagged 462 violations	459 were correct usage
"Mutations tripled"	one document supplied 555 of 788
An edit confirmed on two same-direction reads	contradicted by the third; two falls happen ~25% of the time by chance
A step in the approved plan	went nine cycles unrun — then the resulting noise was diagnosed as a limit of the design
The tool built to enforce the confirmation rule	reported CONFIRMED on its first run, by pooling two different reviewer versions into one baseline

That last one is the whole report in miniature. The instrument written specifically to stop us confirming things too early nearly confirmed something too early, because it mixed two measuring devices and inflated the baseline's spread from 6 points to 22.

None of these were detectable by looking at the number. They were detectable by looking at the shape of the number — which document contributed it, which instrument produced it, whether the median agreed with the total. That check now runs on every rate we report, and it has reversed the conclusion four times in the last three cycles.

The compiler and the spec were the easy part.

Where it stopped, and why we are saying so

Cycle 15 recorded a local optimum. Six consecutive decisions read: weak-keep at a third of a standard deviation, keep-and-watch later reverted, revert, revert, and two holds. The four rules at the top of the evidence table have absorbed six edits between them and return at the same rates. What is left there is semantic damage — conditions weakened, lists summarised, qualifiers dropped — that the compiler has not fixed from instruction in six attempts.

Two of those cycles closed off an approach on evidence rather than on taste, which is the part we would keep. The loop's own rules now carry a change trail for the same reason: an audit found the layer enforcing "one edit, one test, one record" on the spec had itself changed ten times with no record at all, and had once been cited as authority for a rule that did not exist.

The objective rotated, and the loop continued

That is where this report ended when it was first written, and the local optimum was real. What the ending missed is that a local optimum belongs to a loss function. The fifteen cycles above were one run — call it fidelity, how faithfully a compiled document preserves its source — and cycle 15 exhausted it. They had said nothing about how short the document is, which is the property the representation exists to buy.

So the outer loop had one more move in it than the inner one: change the objective, and reset the evidence. A rate measured under one loss function is not evidence about another [M.EVOLVE.2], so when the objective rotates the entire finding pool is discarded and refilled. This is expensive in exactly the way rule-invalidation was expensive in the fidelity run, and correct for the same reason. Three objectives have run since — fidelity (closed at the local optimum above), runnability (one cycle, reverted), and structured-compression, which is open and six batches deep. Cycle 16 restarted at batch 1, restoring the relation operators the legacy grammar had let fall to five of fourteen, and adding hierarchy (R.STRUCT.5) and file-tree references (R.REF.1).

The rotation exposed something the fidelity run had been getting wrong for seventeen cycles without noticing. That run's own rule said to repair an under-applied rule with an example, not by editing the rule — and the executing layer had ignored it every single time. Ninety-six percent of its edits landed at loci the compiler, not the reviewer, was responsible for; all thirty-two were rule edits; not one was an example; and the grammar doubled from twenty-nine rules to forty. A suggestion the layer is free to decline is not a constraint. So it was made arithmetic: check_repair_class now reads the class majority at a locus and refuses a rule edit at any locus that is ≥70% the compiler's fault, routing it to an example instead. Every edit in the compression run is an example, because the code will no longer let it be anything else.

batch	edit	outcome
b1	a rule (relation-operator guard)	REVERT — later found wrong: it worked on 94% of documents and ran away on the other 6%
b2	first example, no new rule (`R.DROP.1`)	keep-and-watch — duplication 73% → 35%, later generalised off its own sample
b3	example (`R.SYM.3`)	edit
b4	example (`R.STRUCT.5`)	reverted — its WRONG side was well-formed line by line, so the compiler read it as a pattern to emit
b5	example (`R.KEEP.6`)	keep — the first fabrication fix
b6	example (`R.KEEP.2b`)	keep — the intended deletion was refused by the parser

Two of those rows are the fidelity run's lessons recurring in the new medium. The b1 revert was scored as a failure of the rule; it is, on re-reading, a failure of the scoring — the rule was right for 94% of the corpus and wrong only on a 6% tail, so the honest verdict was keep-and-watch with a targeted ablation, not revert. It belongs on the same register as the fidelity run's instrument failures, not among its spec failures. And b4 is "it executes shape, not scope" returning to bite the examples. The R.STRUCT.5 example's WRONG side showed three lines each valid in isolation; the compiler emitted the block as a template and produced the defect on eight documents. That retested the meta-rule the fidelity run had written for rules — a WRONG side must be wrong on sight — and confirmed it governs examples too. The headline compression moved to 1.225× that batch, but pooled literal recall fell to 88.8%: a gain we flagged as possibly bought by dropping content, not by encoding it tighter [M.LOSS.3, watch].

The first fabrication the loop could name

Batch 5 was declared an epoch restart, because three instruments moved at once and a rate cannot bridge that discontinuity. The reviewer was reframed — severity and confidence became filing options, a motivation section was added, and V.PRIORITY.1 now ranks compression first; the compiler grammar's motivation was edited to say, in as many words, that invention is allowed; and the meta layer — the one the fidelity run had finally brought under a change trail — was given its own goal and motivation. The old b2–b4 series ends here; b5/b6/b7 is a new one.

Moving the reviewer under the same edit that moves the objective would confound both, so the reviewer was measured alone first: b4's stored encodings were re-reviewed under both reviewer versions, same documents, zero sampling noise. Declined-to-file fell 38 → 32, compression-primary findings rose 17 → 21, high-and-critical rose 45 → 60, and the token budget held. Keep, on its own evidence, before any new batch was drawn.

The edit itself was the loop's first correction of a fabrication — the compiler stating something the source never does. Reading all fifty reviewer overviews together with the frozen analyst surfaced what the ranked pool could not: invented NEVER: lines, invented requirement versions, invented paths. The existing R.KEEP.6 forbade only an invented reason; fabrication is a fidelity defect, the top-ranked term of the new objective. Because the locus is the compiler's, check_repair_class refused a rule edit and routed it to an example.

{ "run": "structured-compression", "batch": 5, "locus": "R.KEEP.6",
  "op": "APPEND", "kind": "EXAMPLE", "class": "FIDELITY", "diff_tokens": 87,
  "linesearch": "3 fixed / 2 unchanged",
  "outcome": "KEEP",
  "result_note": "invented NEVER dropped on swift and generalised (post-to-xhs, flutter); invented frontmatter dropped on playwright; data-stats' invented runtime version stuck — the one miss" }

The meta layer took an edit of its own: M.LOSS.4 now says to read the mode across the whole batch, because the clearest fixes here were on cases the reviewer never filed, which no per-case read can hold. The batch closed with the compression ratio at 1.156×, pooled literals at 91.4%, and reference recall at 94.3% — which answers b4's open worry directly: fidelity came back and the ratio still settled below b4's content-bought 1.225×. Heads advanced to grammar 266f4469, meta d7b2ff07.

The delete the parser refused

Batch 6 first confirmed b5 the way the loop confirms anything — on a fresh batch. The correction took: the frozen analyst now files invented NEVER: lines and fabricated reference paths as conformance failures against R.KEEP.6 by name. A gap in the specification had become a nameable rule.

The batch was supposed to remove something. Five straight batches of additions — one reverted rule, then four examples — is exactly the accumulation M.SHRINK.1 was written against: a loop that only adds is accumulating, not searching, the diagnosis behind the fidelity grammar's 29-to-40 bloat. The candidate was R.SYM.2: zero findings in six batches, and a restatement of R.SYM.1's instruction not to invent operators or tags. This is the one case the fidelity run's warning does not cover — not a rule dead by reference but live by effect, just redundant. It still could not be deleted. R.SYM.2 sits at the end of its section, and the rule parser folds a following ## heading into the preceding rule's body, so removing R.SYM.2 would slide ## TERMS into R.SYM.4, change that rule's hash, and stale its fifteen skills of live evidence. A rule on a section boundary cannot be cleanly deleted. The shrink-debt carried to b7.

So the edit went where the starvation guard pointed instead [M.EVIDENCE.1]: R.KEEP.2b, the highest-rate locus never once edited — 18.0% over 54 skills across six batches — and the worst defect class in the pool, a mis-attached qualifier. Its unexampled sub-mode is an IF: guard indented under the DO: it is meant to gate, so the action fires unconditionally. One example (append, 75-token diff). Line search passed, five fixed to two unchanged: the exemplar slide reordered DO: expand / IF: sentence into IF: sentence / DO: expand, and longbridge, socratic (0 → 8 correct pairs) and a Chinese-language paper skill (0 → 17) followed; meeting-brief was unchanged.

This one came with its own warning built in. Unflagged IF: lines roughly doubled across the batch, 31 → 60 — the exact over-application shape M.FORM.1 exists to catch — but total encoding characters fell 2.7% over the same batch, so the compiler is expressing conditional logic more compactly, not padding. That distinction is now written down: M.FORM.1 judges an example's large behavioural shift by the direction of the character and fidelity change, not the size of the shift — which retrodicts cleanly to why b4's R.STRUCT.5 was correctly reverted and this one correctly kept. Heads advanced to grammar 790cad20, meta b9819c72; the batch closed at 1.125× (a lower ratio, but on a visibly harder sample), literals 94.2%, references 90.7%.

What the new era has not settled

Six batches in, the honest state is a set of tensions the loop has surfaced but not resolved. Compression is not climbing monotonically, and we will not read the ratio as if it were. 1.225× → 1.156× → 1.125× looks like progress only until you note that the first figure was bought by lost content, the last fell on a visibly harder sample, and reference recall slipped from 94.3% to 90.7% across the very step where literal recall rose. Each number is confounded by the one next to it. The shrink-debt is the plainer problem: five examples added, nothing removed, and the one obviously removable rule blocked by a parser boundary rather than by any evidence that it earns its place. A loop built to search is, this era, only adding.

Two things are load-bearing and worth stating flat. The first is that check_repair_class is the only reason any of the above is examples rather than seventeen more rule edits; the constraint that matters is the one the executing layer cannot decline. The second is the division of labour among the heads. They now read: grammar 790cad20, forty rules; reviewer 53126463, eighteen rules, frozen since b5; meta b9819c72, thirty-seven rules; and the analyst fb7138a9, zero rules, ruleless and frozen by design. The analyst — high-temperature, never edited — is the only instrument that has produced genuine cross-batch replication, and the source of new defect shapes, precisely because it encodes no selection. Rules encode what to keep; they do not generate what to notice. The 883 findings now in the pool are still dominated, at 73%, by one duplication defect at R.DROP.1 that is the compiler's to fix and no rule edit has moved — the residue that survived the objective rotation intact.

Method and limits

Batches of fifty documents, drawn disjointly from one corpus, compiled by a small model at temperature zero and reviewed by a larger one. One domain, one corpus, one operator.

Fifty is too few to hold out a validation set: the defects that matter occur at 4–20%, so a split large enough to see them is most of the batch. We spend the sample on the same-batch re-run instead, where the measurement carries no sampling noise, and accept that this is training-set performance and proves less.

The important limit is that we have never tested whether any of it helps. Nothing here measures an agent using a compiled skill against one reading the original — and one paper in our own research index found a re-encoded text scoring below no text at all. Every number in this report is a proxy, measured against a representation whose value to a reader is undemonstrated. That experiment is one batch of work, it is designed and pre-registered, and it outranks another edit to the spec. It is the next thing we run.

We claim no outcome for the compiled representation, in search quality or anywhere else. What we claim is narrower: a specification can be improved against a corpus with a record and a rejection path, the failure modes are in the measurement rather than the spec, and the loop will tell you when it has stopped working if you build it to be able to say so.