Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Graphic design is a domain where no automated oracle can tell you whether a design is good. That gap is the central problem this work addresses, and the answer it proposes is not to train a better model but to evolve the context around a frozen one.

The system wraps a frozen frontier language model that controls equivalents of Photoshop, Illustrator, and InDesign through more than 230 tools. Around that fixed model sits an external library of natural-language skill files — Markdown playbooks, each describing a reusable design procedure. The library evolves along two axes: widening mints new skills when a recurring subtask has no coverage, and deepening rewrites existing skills that keep appearing in failed trajectories. Neither axis ships changes freely. A matched replay gate re-runs both the candidate and the incumbent against the same frozen contexts and accepts a change only when it wins at least one replayed case without losing any — a conservative stance borrowed from safe policy improvement.

Across five rounds of evolution over user-traffic briefs, with no weight updates and no human reward labels, the skill bank grows from 76 documentation-seeded skills to 139. The ablation result is the most instructive finding: the documentation-derived cold start does not improve over the no-skill baseline at all. Widening alone and deepening alone each produce modest gains, but their combination yields a superadditive jump in task completeness. The paper's explanation is structural — minted skills need refined retrieval descriptions to surface reliably, while deepening reroutes unfixable failures back into the widening pool.

The paper is honest about where the approach breaks down. A natural-language skill can describe a procedure but cannot override a model's strong default behavior without also scoping the available tool set. Fine geometric operations remain limited by VLM perception, and long skills lose fidelity as the frozen model stops following the script partway through. The personalization appendix makes this concrete: a recursive Droste effect remains imprecise even with a dedicated skill, and a complex tessellation pattern degrades at every stage of distillation. The authors name a related failure mode "vibe drawing" — the agent produces something that resembles the target without being the actual usable artifact.

The replay gate's design is worth attention for anyone building deployed agents. Rather than comparing absolute scores, which drift across VLM judge runs, it holds context fixed and compares arms pairwise within each context, eliminating judge drift and upstream-state confounding in a single move.