onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

The core insight behind onPanda is simple but underexploited: when a model generates a mostly-correct response, the cheapest fix is to intercept the first wrong token rather than rewrite the whole thing. The annotator reads the output, spots the first bad token, clicks a replacement from the model's own top-candidate list, and lets the model continue from that corrected prefix. The loop repeats until the response qualifies. That's it.

What makes this more than a UX trick is what it preserves. Because the model regenerates everything after each correction point, the vast majority of tokens in the final response come from the rollout model itself. At production scale across vision and audio sessions, 97% of tokens are model-generated, with manually typed text accounting for under 1%. That on-policy fidelity matters: perplexity under the rollout model stays within 1% of the re-sampling baseline for onPanda, versus a substantially higher deviation for manual post-editing.

The controlled study is modest—three annotators, 21 prompts, one rollout model—but the numbers are directionally clear. Median annotation time drops by roughly 52% compared to post-editing with POTATO, while pairwise win rate (judged by GPT-5.5) favors onPanda at about two-thirds of comparisons. Argilla's preference-ranking workflow matches onPanda on speed at the median but balloons in mean time on hard prompts, and it only yields a qualified SFT response on about half the prompts. OnPanda reaches full SFT coverage by design, since annotators keep correcting until the response qualifies.

The supervision signal itself is the more interesting contribution. Every correction records the error position, the rejected token, and the chosen replacement—naturally paired positive and negative samples at token granularity. The paper argues this is strictly finer than response-level preference data and can feed reward model training, DPO, or process reward models without additional labeling. Whether that downstream training benefit actually materializes is explicitly left unverified; the authors flag it as future work.

For agent settings, the response template mechanism deserves attention. Reasoning chains, tool-call arguments, and special tokens are rendered into the model's native token stream so annotators can correct them directly. Tool calls can be held for approval, corrected before execution, or rejected with guidance—rejected trajectories are automatically retained as negatives. This makes interactive trajectory annotation in real environments tractable rather than theoretical.

The Panda-CVL benchmark reveals how hard the underlying task actually is for current models. The best F1 on token-level correction reaches only around 17%, with error localization remaining weak even for models that reliably follow the output format. That gap between format compliance and genuine correction ability is a useful calibration for anyone building automated annotation pipelines.

The tool requires prefix-continuation and logprob support from the inference API—a real constraint for some proprietary providers, though the paper notes it is satisfied by several major vendors and all mainstream open-weight serving frameworks.