Comparing a skill's claims to its code lifts misalignment detection from 0.45 to 0.89 Macro-F1

Notes on Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach (arXiv:2607.10534) — Chengjun Zhang, Yang Gao, Jianna Hur, Jingjing Zhang, Sagar Samtani · July 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security

SkillsMP, the largest open-source Agent Skills marketplace, supplied a corpus of 264,937 normalized skill packages out of 273,657 catalog entries, each split into three layers: metadata (name, description, tags), instructions (the SKILL.md procedural text), and resources (scripts, configs, dependencies). Divergence between what the metadata claims and what the instructions and resources actually do — cross-layer misalignment — is what PL-HCL (Progressive Loading-Aware Hierarchical Contrastive Learning) is built to catch before a skill ever runs. Training mirrors how a skill actually loads: first continued pretraining on metadata plus instructions alone, then on full packages including resources. On top of that sits a contrastive learning stage, trained against two kinds of synthetic negatives per skill — swapping in another skill's metadata, and perturbing the skill's own metadata — while the true metadata-instructions-resources triple stands as the positive example the model learns to recognize as consistent.

An unadapted 8B model asked to flag misalignment is close to useless: zero-, two-, and five-shot prompting on plain Llama-3.1-8B and a cybersecurity-tuned Foundation-Sec-8B both land around 0.45-0.46 Macro-F1, with F1 on the misaligned class as low as 0.03 — the models default to calling almost everything aligned. Continued pretraining alone barely moves that: Macro-F1 plateaus near 0.48-0.52 even after the model has absorbed skill-package vocabulary and structure. Adding the contrastive stage is what closes the gap: Macro-F1 jumps to 0.872 (Llama-3.1-8B) and 0.889 (Foundation-Sec-8B), with misaligned-class F1 rising to 0.79-0.82. The gains hold on a separate, human-verified challenge set the model never trained on directly — every contrastive pair it learned from was synthetic. Two qualitative examples make the mechanism concrete: a skill describing itself as a plain assistant while shipping unrestricted Bash access and command-injection instructions flips the model's aligned/misaligned logit gap from +4.33 to -0.13 after PL-HCL; a skill claiming to be a narrow TypeScript utility while its body runs an external curl call flips from +5.81 to -1.50 — no malicious signal required, just a description that doesn't match the code.

Key numbers

Training corpus (SkillsMP marketplace)	264,937 normalized skills (of 273,657 listed)
Unadapted baseline (zero/few-shot prompting)	Macro-F1 ~0.45; misaligned-class F1 as low as 0.03
Continued-pretraining-only plateau	Macro-F1 ~0.48-0.52
Full PL-HCL, best backbone (Foundation-Sec-8B)	0.937 acc / 0.817 F1_misaligned / 0.889 Macro-F1
Human-verified challenge set	1,444 skills (1,150 aligned, 294 misaligned)

What this means for skill authors

The challenge-set breakdown is the detail worth sitting with: of the 294 skills human annotators flagged as misaligned, 196 were independently judged safe and only 60 malicious. Misalignment and malice are different axes — a skill can describe itself accurately and still be dangerous, or describe itself inaccurately and still be harmless in effect — and this framework is built specifically to catch the second case, which intent-focused security scanning has no reason to flag because nothing in it looks hostile.

Baseline numbers matter more than the eventual 0.89 — they set the honest floor for what an off-the-shelf LLM judge can do here without dedicated training. Sub-0.15 F1 on the misaligned class, even from a model purpose-built for security work, isn't a screening tool — it's noise with a plausible-sounding rationale attached. Continued pretraining on skill text alone doesn't fix that either, which is the more useful negative result: a model can learn what skill packages look like without learning to compare what one layer claims against what another layer does. That comparison has to be trained for directly.

For anyone running a skill catalog, the practical reading is that description-versus-code consistency deserves to be checked as its own axis, separate from intent or safety scanning — the same logic behind pairing an LLM's read of a skill's stated purpose against a separate read of what it actually does before anything reaches a catalog. At the scale of a single open marketplace (264,937 packages), manual review of that gap isn't an option; a pre-execution classifier that routes likely mismatches to closer inspection is the realistic middle ground between trusting the README and reading every script by hand.

Skills related to this research

Mermaid To Gif Mermaid to GIF transforms Mermaid diagram code into animated GIF files with four distinct animation styles tailored to different diagram types. Extract diagrams from .mmd files or markdown code blocks, then customize animation, theme, and rendering options to suit your presentation or documentation needs.★ 0 geo-technical geo-technical helps you evaluate your website's technical foundation through the lens of both traditional search engines and AI-powered platforms like ChatGPT, Claude, and Perplexity. It identifies gaps in crawlability, indexing, and content structure that matter to modern discovery systems. Use this skill to shift your optimization strategy toward where traffic is actually heading.★ 9,138

Related notes

References

Zhang, C., Gao, Y., Hur, J., Zhang, J., & Samtani, S. (2026). Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach. arXiv:2607.10534.
ReAct — reasoning-and-acting framework for LLM tool use, cited as related work on tool-augmented agents.
SimCLR — contrastive representation learning framework, cited as foundational to the paper's contrastive objective.
CLIP — multi-modal contrastive learning framework, cited alongside SimCLR as related contrastive-learning work.
Domain-/task-adaptive pretraining (DAPT/TAPT) — continued-pretraining methods for domain specialization, cited as the basis for the paper's two-stage CPT curriculum.