jev-align builds a real deploy-capture-label-improve loop for LLM classifiers

Active learning for LLM-based classifiers has been a research problem for years; jev-align is a practical CLI that closes the loop between uncertainty sampling, human labeling, and prompt optimization in a single terminal workflow. The mechanism is worth understanding precisely: each round it evaluates your dataset, identifies the rows the model is least certain about, pulls those plus a random audit sample for you to label, then runs GEPA—a gradient-free prompt optimization method—against your accumulated labels and optional rationales. You see the score, the certainty delta, and a diff of the proposed definition change. You accept or reject it. Nothing advances automatically, regardless of how high the training score climbs.

The task coverage is broader than most labeling tools bother with: binary, multiclass, multilabel, and ordered rubric scoring are all first-class. Configuration is either guided or fully flag-driven, so it fits both exploratory and scripted use. The default pool is the first 1,000 rows of a CSV, Parquet, or JSONL file; you can tune annotations per round (5, 10, 15, or 20), hold out 20% for evaluation, and set GEPA's metric-call budget, which defaults to 300.

The production integration is the part that distinguishes this from a one-shot labeling session. You load a saved function in your application with capture=True, and it silently collects real examples. Later you resume the run, label what was captured, and GEPA proposes the next version. That feedback loop—deploy, capture, label, improve—is the actual pitch, and it's architecturally sound even if the implementation is early.

The reflection model (the one GEPA uses to rewrite the prompt) is decoupled from the Jev evaluation model. OpenAI, Anthropic, and Gemini are auto-detected; anything LiteLLM supports, including local vLLM endpoints, can be passed via --reflection-model. That separation matters: you can run a cheap local model for reflection while the production classifier uses a hosted API, or vice versa.

The public registry at ai-functions.dev lets you publish a function—definition, input signature, labels, rationales, splits—and others can pull it without signing in. The README is explicit that unpublishing removes a function from discovery but cannot revoke copies already downloaded, and that the registry makes no claim about whether annotations were human-reviewed. That honesty is appropriate given the tool's experimental status.

The main caveat is that GEPA's quality depends entirely on the quality of your labels and rationales. The tool surfaces uncertainty well, but it cannot compensate for noisy or inconsistent annotation. For teams without a clear labeling standard, the guided review step is load-bearing, not optional.