Conditioning the perception latent on the text skill card cuts cross-modal redundancy 9x — and gets 2.3x the step-consistency of 5-sample self-consistency at roughly the same latency as 1 sample

Notes on Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck (arXiv:2605.08526) — Zihan Huang, Junda Wu, Tong Yu, Qianqi Yan, Rohan Surana, Uttaran Bhattacharya, Lina Yao, X. Wang, Julian J. McAuley · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Frontier & other

Agents built on vision-language models rarely repeat themselves. Ask the same policy to complete the same web task twice and the click sequences drift, even though the underlying reasoning hasn't changed. The standard fix is self-consistency — sample several rollouts, vote on the majority action — which trades inference cost for stability and throws away the pattern it just discovered. A reusable agent skill splits that task-relevant signal differently: some of it verbalizes cleanly into a procedure card, some of it lives in perceptual detail that resists description. Storing both streams side by side, uncoordinated, just duplicates whatever the text already captured.

Conditional Multimodal Information Bottleneck (CMIB) factors a single joint information bottleneck over the skill into two sequential stages: a text-stage bottleneck that compresses trial-and-error trajectories into a short, interpretable skill card, and a second bottleneck over a multimodal latent vector that's explicitly conditioned on that card — so it only has to encode residual visual information the text didn't already cover. A frozen LLM produces the card under a length-and-utility tradeoff; a variational encoder/prior pair (a Q-former plus MLP feeding a frozen Qwen2.5-VL-7B backbone) trains the latent and fuses it in as a soft prefix. Nothing in the base model gets updated. Tested on Multimodal-Mind2Web and ScreenSpot against a vanilla agent, a text-only skill card, and self-consistency up to 5 samples, CMIB pushes step success rate up while making repeated trials converge on the same actions — at close to single-sample inference cost.

Key numbers

Step Consistency, CMIB vs. 5-sample self-consistency	41.44% vs. 17.89%
Avg. Step Success Rate, CMIB vs. Qwen2.5-VL-7B baseline (Mind2Web)	38.7% vs. 30.4%
Per-step latency, CMIB vs. 5-sample self-consistency	3,510 ms vs. 17,681 ms
Cross-modal redundancy I(c;z), CMIB vs. unconditioned two-stream	0.18 vs. 1.69
ScreenSpot avg. action success, CMIB vs. Qwen2.5-VL-7B baseline	57.9% vs. 53.0%

What this means for skill authors

The ablation is the part worth sitting with. Dropping the conditioning constraint — letting the text card and the perception latent be encoded independently instead of one conditioned on the other — doesn't just add noise, it multiplies measured redundancy between the two streams by roughly 9x (0.18 to 1.69) and step success drops from 41.66% to 39.18%. Removing the latent entirely and keeping only the text card drops it further to 37.95%. That's a clean demonstration that a skill's perceptual half isn't free-floating context to bolt on — it needs to know what the text half already said, or it just re-encodes it.

For anyone authoring or packaging agent skills, this is a concrete argument against the two-file pattern — a markdown procedure plus a loose folder of screenshots or embeddings — unless the two are built to reference each other's content rather than duplicate it. A vetted catalog that only inspects a skill's text — the card, the prompt — is structurally blind to exactly the residual failure mode isolated here: a visual component that silently repeats or contradicts its own procedure text. Checking whether the two modalities agree, not just whether each looks reasonable on its own, is the qualitative case for treating a skill as a checked pair rather than a single artifact.

The efficiency framing also reframes what a 'skill' should mean at inference time. Self-consistency treats reliability as something purchased per call, via repeated sampling; CMIB treats it as something built once into a reusable representation and paid for at construction time instead. That's the same bet skill reuse generally makes — amortize the expensive part once, retrieve cheaply after — except now it comes with an information-theoretic argument behind it, not just an engineering convenience one.

Skills related to this research

Baoyu Image Gen Baoyu Image Gen creates images from text prompts using OpenAI, Google, or DashScope providers. Configure default settings, choose quality levels and aspect ratios, and optionally use reference images to guide generation.★ 331 10 Music Video This skill transforms your music into precise visual prompts for Seedance 2.0 on Higgsfield, a platform that synchronizes generated video to your uploaded audio track. Learn to reference specific beats, describe genre-matched visuals, and apply proven hooks—from particle explosions on drops to color shifts on harmonic changes—that lock your visuals to every moment of your song.★ 712 mermaid-visualizer Mermaid Visualizer transforms written content into polished diagrams ready for presentations and docs. It handles process flows, system architectures, comparisons, mindmaps, and sequence diagrams while automatically preventing common syntax errors. Choose your layout, detail level, and style—the skill generates clean, rendering-ready code for Obsidian, GitHub, and other Mermaid platforms.★ 4,045 Canghe Image Gen Canghe Image Gen creates images from text prompts across multiple AI providers including OpenAI, Google, DashScope, and Canghe. Configure your preferred provider, quality level, and aspect ratio, with support for reference images and batch generation when needed.★ 418

Related notes

References

Zihan Huang, Junda Wu, Tong Yu, Qianqi Yan, Rohan Surana, Uttaran Bhattacharya, Lina Yao, Xin Eric Wang, Julian McAuley, "Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck," arXiv:2605.08526 (2026)
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, Yu Su, "Mind2Web: Towards a Generalist Agent for the Web," NeurIPS 2023
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu, "SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents," ACL 2024
Naftali Tishby, Fernando C. Pereira, William Bialek, "The Information Bottleneck Method," arXiv:physics/0004057 (2000)
Xuezhi Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models," arXiv:2203.11171 (2022)