Simple Jev skips text generation entirely and reads logits to classify

Simple Jev sidesteps the usual generate-then-parse loop entirely. Instead of asking a model to produce a JSON completion and then extracting the answer, it reads the next-token logits directly for a predefined set of answer labels and builds the response from those scores. No decode loop, no output tokens — usage.output_tokens is always zero.

The three question types cover most structured-decision needs. choice picks the highest-probability candidate from 2 to 50 options and returns the full distribution. score computes an expected value over an ordered rubric, yielding fractional results — a three-level rubric returns values between 0 and 2. noul derives a truth/support judgment from 0.01 to 0.99 by aggregating over nine rating tokens. The README is explicit that these distributions are not calibrated probabilities of correctness, which is an honest and important caveat.

The more interesting engineering is the shared-prefix KV cache. When a single request contains several questions about the same context, the server tokenizes each question's full prompt, finds the exact common token prefix, runs one prefill pass, saves the KV cache, then batches the per-question suffixes against that cache. The README illustrates the arithmetic: four questions each with a 1,000-token shared prefix and a 50-token suffix require processing 4,200 tokens if evaluated separately, versus 1,200 tokens with prefix reuse. That is a concrete reduction in repeated computation, though the README is careful to note that cache copying, padding, and suffix attention still carry costs — this is not a measured latency ratio.

Cache reuse currently lasts only within a single request, and the server processes model requests serially. The HF implementation runs locally with Transformers and PyTorch; a public demo API is available with a 2k-token context limit and a rate cap of 2 requests per second. Compatibility is not guaranteed across all open models — labels must each extend the rendered prompt by exactly one distinct token, and the server validates this.

The RFDT component adds fine-tuning on top: train a smaller model directly on answer-token logits using the same prompt structure, with optional teacher labeling from a larger model. LoRA adapters and multi-GPU training are supported. Hosted fine-tuning on Featherless is described as forthcoming.

For agent pipelines that need fast, deterministic classification — routing, rubric scoring, fact support judgments — this approach trades generation flexibility for speed and structural guarantees. The tradeoff is real: model capability and question wording still determine answer quality, and the server's clean response structure says nothing about whether the underlying decision is correct.