skillfed

Pre-simulated skill libraries beat live code generation 44.7% to 28.7% -- at zero runtime cost

Notes on Skill Discovery for Software Scripting Automation via Offline Simulations with LLMs (arXiv:2504.20406) — Paiheng Xu, Gang Wu, Xiang Chen, Tong Yu, Chang Xiao, Franck Dernoncourt, Tianyi Zhou, Wei Ai, Viswanathan Swaminathan · April 2025 · Conference of the European Chapter of the Association for Computational Linguistics

Note published · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

This framework skips runtime code generation entirely: it pre-builds and validates a library of Adobe Illustrator scripts offline, then serves them by lookup instead of writing ExtendScript on the fly. Task ideas come from two directions. A top-down pass walks documented software functionality -- drawing, arranging objects, and so on. A bottom-up pass mines an API synergy graph -- built from which Illustrator APIs actually co-occur in working scripts -- using GNN link-prediction to propose tasks for API pairs the system has never seen paired before. Each candidate then runs through skill generation with trials: an LLM writes a script, an LVLM validator checks both the execution output and the resulting canvas state, and the LLM gets up to three offline attempts to fix what broke -- all before any end user asks for it.

On a 94-task held-out set, plain retrieval against the pre-built skillset succeeds 44.7% of the time, in 0.1 seconds, for zero runtime tokens. Live gpt-4o code generation manages 28.7% success at 4.0 seconds and 666 tokens. A retrieval-augmented generation variant -- stuffing retrieved skills into the prompt rather than running them directly -- lands in between at 42.6% success, 4.3 seconds, 1219 tokens. The synergy-graph approach earns its complexity: Hit@5 for predicting which APIs pair well reaches 37.3%, more than double the 16.7% a plain semantic-similarity baseline gets, and bottom-up exploration built from it touches 151 distinct APIs versus 49 for top-down and 48 in Illustrator's own sample scripts. None of this comes free on the first try -- a single generation pass succeeds only 16.7% (top-down) to 23.1% (bottom-up) of the time, roughly doubling to 34.9% and 46.6% by the third offline trial.

Key numbers

Retrieval-only success rate vs. live gpt-4o generation44.7% vs 28.7%
Response time / runtime tokens: retrieval vs. live generation0.1s, 0 tokens vs 4.0s, 666 tokens
Single-attempt script success rate before offline refinement16.7%-23.1%
Distinct APIs covered: bottom-up vs top-down vs native samples151 vs 49 vs 48
GNN link-prediction Hit@5 vs semantic-similarity baseline37.3% vs 16.7%

Skills related to this research

audio-router audio-router intelligently directs audio processing requests to the right specialized skill, whether for playback control, sound analysis, or reactive audio handling. Built for Claude, it streamlines how audio workflows are orchestrated across multiple focused capabilities.★ 8 idea-generator Idea Generator walks founders through structured discovery to uncover product concepts grounded in what they already know or do. It captures context through targeted questions, synthesizes patterns into candidate directions, scores each against five key axes, and writes a shareable product-idea.md document that feeds into downstream planning skills.★ 168 Tanstack Query This skill provides structured guidance on implementing TanStack Query patterns across React applications. It covers query key design, caching strategies, mutations, error handling, prefetching, infinite queries, SSR integration, and offline support through prioritized rules with code examples.★ 201 Figma Generate Library Build production-ready design systems in Figma by orchestrating multi-phase workflows that extract tokens from code, create variable foundations, construct components with proper variant sets, and document design principles. Works alongside the figma-use skill to enforce quality patterns from real-world systems.★ 1,925
DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not… He et al. · similarity 0.84

Related notes

References

  1. Xu, P., Wu, G., Chen, X., Yu, T., Xiao, C., Dernoncourt, F., Zhou, T., Ai, W., & Swaminathan, V. (2025). Skill Discovery for Software Scripting Automation via Offline Simulations with LLMs. arXiv:2504.20406.
  2. Kipf, T. N., & Welling, M. (2017). Semi-Supervised Classification with Graph Convolutional Networks. ICLR.
  3. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291.
  4. Wang, B., Fang, H., Eisner, J., Van Durme, B., & Su, Y. (2024). LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error. ACL. arXiv:2403.04746.