Semgrep, refinement types, and Z3 turn a "formally verified skill" into a proof you can re-check yourself — minus one named gap

Notes on Methods for Formal Verification of Agent Skills: Three Layers Toward a Mechanically Checkable Capability-Containment Proof (arXiv:2605.23951) — A. Metere · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security

Agent skills split into two halves: a deterministic script side that runs known code, and a non-deterministic LLM side that decides what to call next. This paper treats capability-containment — proving a skill's tool calls never exceed what its manifest declares — as a property of that whole system, not just the script, and closes it with three methods built to compose rather than compete. First, abstract interpretation walks a capability-effect lattice to over-approximate what the script's code can actually do, reusing existing dataflow engines (Semgrep, CodeQL, Pyright) instead of building new analyzers from scratch. Second, the tool-call dispatch function gets wrapped in a refinement type that mechanically rejects any call whose inferred capability falls outside the manifest's declared set, no matter what the LLM asks for. Third, SMT-bounded model checking with Z3 tests the parent paper's biconditional correctness criterion against a session-length horizon, so any violation surfaces as a concrete counter-example trace rather than an abstract failure.

Together the three layers cover the threat model soundly, but the authors name one residual themselves: an LLM can always refuse to act, and neither the static analysis nor the bounded proof forecloses that. It's caught only at the session boundary, by the runtime's own biconditional check — not by the formal layers. A worked example makes the gap the method does close concrete: a skill that summarizes fetched HTML declared only network-egress and cache-read capabilities, and the static-analysis layer flagged an undeclared filesystem-write effect on the cache directory that the manifest never listed. After the manifest was corrected, the refinement-type layer passed and the bounded model checker returned unsat for sessions up to 100 tool-call envelopes — no containment violation exists within that bound. The full toolchain, three checkers plus a bundle producer and re-checker, ships as zero-dependency JavaScript inside the open-source enclawed framework, backed by a 53-test unit suite and an end-to-end CLI demo.

Key numbers

Composable verification layers	3
Unit tests passing in the enclawed suite	53
SMT bound cleared with zero violations (worked example)	100 envelopes
Named residual left uncovered by the proof	1 (LLM refusal to act)
Typical SMT parameters solved in seconds	\|D\| ≤ 10, K ≤ 8

What this means for skill authors

For skill authors, this is what "formally verified" should mean instead of a badge: a re-checkable evidence bundle — static-analysis output, a type-checker discharge, an SMT certificate, all signed — that a downstream consumer can reproduce without trusting whoever produced it. That distinction matters more than which specific tools are named. Any catalog that claims to vet skills eventually has to answer whether "vetted" means "we ran a scan at intake" or "here is a proof you can independently re-derive," and this paper draws that line precisely enough to build against.

The worked example matters more than the theorems for practitioners: a manifest that looked complete on inspection had an undeclared filesystem-write effect that static analysis caught automatically. That's exactly the gap dual-tool scanning approaches — SkillFed's included — exist to surface: the space between what a skill's manifest claims and what its code actually touches. A formal proof doesn't replace that kind of scanning; it's what scanning graduates into once you bound the search space and sign the result.

The honest caveat is one the paper states outright: none of this constrains whether the model decides to act, only what happens once it does. A skill can clear all three layers and still stall, loop, or silently decline — the containment proof is silent on liveness. Read "formally verified" as a bound on what a skill can do wrong, not a guarantee that it does anything at all.

Skills related to this research

Daytona Flow Validator This skill validates Daytona flow configurations by checking for syntax errors and structural correctness. Use it to catch configuration problems early and prevent runtime failures in your workflow deployments.★ 1,925 Daytona Recording Artifacts This skill enables you to fetch and work with Daytona recording artifacts—the files and data captured during your recorded sessions. Quickly locate and retrieve stored outputs to review, analyze, or integrate with other workflows.★ 1,925 Daytona Flow Validator This skill inspects Daytona workflow definitions to identify misconfigurations, compliance gaps, and potential runtime errors. It validates your setup against best practices and standards, helping you catch problems early in development. Use it to maintain workflow quality and prevent deployment failures.★ 17,319 Daytona Recording Artifacts This skill enables you to fetch and work with Daytona recording artifacts and session information. Query your recorded sessions to retrieve artifacts and metadata, streamlining access to historical recording data within your workflows.★ 17,319

Related notes

References

A. Metere, "Methods for Formal Verification of Agent Skills: Three Layers Toward a Mechanically Checkable Capability-Containment Proof," arXiv:2605.23951 (2026).
A. Metere, "Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes," arXiv:2605.00424 (2026).
A. Metere, "enclawed: A Configurable, Sector-Neutral Hardening Framework for Single-User AI Assistant Gateways," arXiv:2604.16838 (2026).