Certainty tags turn 0–9% accuracy into 88–100%

Notes on Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills (arXiv:2606.11897) — S. Liu, Jiayao Chen, Chengwei Qin, Yanqing Hu, Jufan Zhang, Linyi Yang · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Notes2Skills runs lab notebooks through two stages before any of it reaches an agent. Stage 1, epistemic directive extraction (EDE), scans free-text notes and tags each statement as FACT, JUDGMENT, or SUGGESTION, then sorts it into one of five directive types: flag data for review, note a condition change, suggest an analysis step, revise a protocol, or adjust a parameter. Stage 2, MetaSkill compilation, turns those labeled directives into agent-loadable Markdown skills, where each one keeps its certainty tag plus a SHA-256 hash pointing back to the exact source excerpt. They validated the pipeline against 461 hand-annotated segments across three corpora — informal free-form lab notes on one end, formal wet-lab protocols on the other.

The best extraction setup — Claude, few-shot — hit an F1 of 0.737 on detecting directives and a quadratic weighted kappa of 0.761 on grading certainty; compilation then preserved every one of 149 finalized directives, labels and source links included, with zero loss. The real test was downstream. Across three nanopore wet-lab sessions (48 files total) and seven pipeline configurations, only the full MetaSkill setup with executor verification avoided both failure modes at once. A baseline that just handed raw notes to an LLM fell into uncertainty laundering — treating tentative judgments as confident actions — and scored 0–9.1% on uncertainty-heavy sessions, versus 88.2–100% for the certainty-aware pipeline. Strip the certainty schema back out and hand the executor authority anyway, and accuracy on a fact-dominated session collapsed from 88.9% to 44.4% — the paper's clearest case of directive loss, where firm author instructions get buried and ignored once everything reads as equally tentative.