SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

Static analysis of generated PLC code is a comfortable lie. Programs that compile cleanly and satisfy property checks can still misconfigure a timer, overwrite a fault default with a low-flow setpoint, or hold an output at its initial value forever. SemaPLC is built around that uncomfortable fact.

The core mechanism is a completion rule: the agent cannot declare a task done. Only logged external verification can. Every edit voids prior verdicts, so the agent cannot bank a passing compilation result and then silently change the logic. Every claimed pass is cross-checked against the tool call log. This delivery-integrity guarantee is what separates SemaPLC from prior systems that use runtime evidence to demonstrate that code can run rather than to measure how reliably it runs.

The evaluation makes the distinction concrete. On a 65-task project-context track—where generated logic must compile and execute inside an existing industrial ST project, not as an isolated function block—the three baselines score between 71.7 and 75.7 on static behavior. That four-point spread looks like rough parity. The dynamic scores tell a different story: baselines range from 22.4 to 31.4, while SemaPLC averages 52.2 and never falls below 30 across seven backbone models. Methods that look nearly identical under static evaluation are sharply separated once execution is the judge.

The case study in the paper makes this visceral. A coking-refinery requirement binds one setpoint to two competing conditions. Two baselines fail to compile at all. Agents4PLC compiles but its low-flow assignment overwrites the fault default—the wrong value, delivered silently. SemaPLC's intermediate candidate also compiles with wrong behavior. The difference is what happens next: runtime validation forces the low-flow input, observes a concrete mismatch, localizes the repair to output selection by cause, and re-verifies both abnormal cases before declaring success. No static check catches this class of defect.

The layer ablation on DeepSeek-V4-Flash quantifies the cost of that reliability. Adding specification, compilation, and runtime checks cumulatively raises the dynamic score from 23.1 to 54.1, while static behavior moves far less. Mean tokens per task climb from roughly 34k to 129k, and requests from 8.9 to 47.8. The dynamic gain is real and expensive, and the paper doesn't obscure either fact.

Two honest limitations are stated. Dynamic scoring covers a bounded scenario set derived from the hidden reference, so behavior under unseen operating conditions remains unmeasured. And the advantage narrows at the frontier: on GPT-5.5, SemaPLC leads Agents4PLC by only 1.8 dynamic points. The harness supplies reliability where the model alone falls short; it is not a fixed margin that persists as models strengthen.

The broader implication for anyone building code-generation agents is blunt: an evaluation that stops before execution cannot separate reliable methods from unreliable ones. Timer behavior, state transitions under competing conditions, and interlock priority are invisible to compilers and property checkers alike. If the domain has a runtime, the benchmark needs one too.