LLM-generated skills move data-science accuracy 1.2 points — same as filler text

Notes on Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows (arXiv:2607.07504) — Wei-Jung Huang · July 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks

The team built one reusable skill file per stage of a data-science agent's workflow — data preparation, data extraction, statistical analysis, and reporting — generating each with Gemini 2.5 Pro and shipping it unedited, mirroring how a low-curation skill pipeline actually gets used. Each skill split into four parts: routing triggers, a core procedure, worked examples, and reference notes. Rather than testing skills as a single on/off switch, they ran a component ablation: five conditions from no-skill up to the full four-part file, plus a token-matched control of irrelevant office-supply text padded to the same length, and a variant instructing the model to prioritize task instructions over skill guidance. Nine model configurations across OpenAI, Google, and Anthropic — spanning compact, frontier, and explicit-reasoning tiers — ran three times each over 56 deterministically verified tasks, yielding 7,560 runs in the main ablation and 10,584 runs total with the supplemental controls.

Nothing beat the no-skill baseline. Pass rates across all five conditions spanned only 1.2 percentage points, and every comparison against the task-only prompt came back with p ≥ 0.396, including on a mixed-effects model with per-task random intercepts. The token-matched filler content scored statistically the same as the real skill (66.9% vs. 67.5%), meaning models showed no visible benefit from the skill's actual content beyond paying a token cost for it. Break it down by stage and three different things are going wrong. Extraction and reporting already sit near ceiling (88.9% and 97.6% baseline) — no headroom for a skill to lift. Data preparation got worse with a skill attached, 57.1% down to 50.8%, because generic cleaning heuristics collided with task-specific instructions. Statistical analysis never moved off the floor (29.4%) no matter which skill condition ran, which points to a reasoning gap that added prompt text can't close.

Key numbers

Spread across all skill conditions	1.2 pp
Weakest p-value across conditions	p ≥ 0.396
Main-ablation runs	7,560
Input-token overhead, Full skill	~4.5×
Human-curated skill gain, prior work (contrast)	16.2 pp

What this means for skill authors

The useful part of this null result is that it's scoped precisely. The paper's own point of comparison — prior work reporting a 16.2 percentage-point gain from human-curated skills against no aggregate benefit from LLM-generated ones — says the failure isn't "skills don't work," it's "unedited, family-level, flatly-injected skills don't work." A single skill trying to cover all of data preparation is generalizing across enough distinct scenarios that its baked-in heuristics will eventually fight the task's actual instructions, and in this study models followed the skill over explicit task instructions often enough to move the failure rate.

Two secondary findings matter more than the headline. The token-matched control landing statistically even with the real skill content is the stronger claim — it suggests the model wasn't extracting anything skill-specific at all, just carrying a fixed token tax. And no component was reliably good: reference notes trended negative in every condition they appeared in, while a priority directive telling the model to favor task instructions over skill guidance recovered some failures and introduced others, netting out near zero.

For anyone assembling a skill catalog, the lesson sits in what wasn't tested here as much as what was: no expert-edited or task-specific positive control was included, so this doesn't say curation is worthless — it says a skill file has to earn its place with a measured lift over no-guidance, not just plausible-looking prose. Vetting that checks whether a skill loads and reads well, without checking whether it beats a token-matched control on held-out tasks, is checking the wrong thing. That's the bar a catalog worth trusting has to clear before calling anything vetted.

Skills related to this research

3dgs-experiment-planner This skill streamlines experimental design for 3D Gaussian Splatting research by organizing a comprehensive catalog of 789+ techniques across 25 distinct categories. Researchers can explore proven methodologies, benchmark approaches, and validate experimental frameworks before submission. The interactive explorer helps identify relevant prior work and establish solid foundations for novel contributions.★ 129 Statistical Analysis Statistical Analysis guides you through test selection, assumption verification, and effect size reporting for academic research. It covers frequentist methods like t-tests, ANOVA, and regression alongside Bayesian approaches, with specialized workflows for survival analysis, count models, and reliability assessment.★ 284 2analysis Modeling 2analysis Modeling decomposes mathematical modeling competition problems into structured components: subproblems, data understanding, assumptions, variables, objective functions, and constraints. It produces a complete modeling report with formulas and algorithms ready for code implementation.★ 2,985

Related notes

References

Wei-Jung Huang, "Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows," arXiv:2607.07504 (July 2026).
SkillsBench (cited in this paper as prior work): human-curated skills produced a 16.2 percentage-point improvement over no-skill baselines, while LLM-generated skills showed no aggregate benefit.