Rizzo-AI-Academy/rizzo-flow

The core insight here is that classification doesn't require generation. When an LLM decides between options, the answer is already encoded in the logits after a single forward pass — the probabilities assigned to each candidate letter. Rizzo Flow reads exactly those logits and nothing else, returning typed JSON with probabilities instead of text you then have to parse and validate.

This mirrors the programming model of TypeSafe's Jev service, which the README explicitly credits as the idea being reproduced. The difference is that Jev is closed and hosted; Rizzo Flow runs on your own hardware via llama.cpp, with open weights (Spark-X2.5-4B or 1.7B, both Apache-2.0), and exposes a compatible HTTP interface so code written against the TypeSafe API can redirect to localhost by changing one URL. The README is clear that this is interface compatibility, not model equivalence — probabilities are uncalibrated by default, and no claim of matching Jev's quality is made.

The architecture is efficient in a specific way: all questions in one request share the state's KV cache. A batch of 21 questions on a roughly 2,000-token state takes about one second on an RTX 5060 Ti at 8-bit quantization, versus one second per question if each were processed independently. The tradeoff is that shared-prefix scoring occasionally flips near-ties — 13 of 777 decisions changed argmax between shared and fresh scoring at Q8_0, all with margins below 0.24. The README's advice is blunt: don't put a threshold near 0.5.

The benchmarks are honest in a way that's worth noting. Against SemIf's committed fixtures, the 4B model at Q8_0 scores 0.812 balanced accuracy on the base set and 0.848 on perturbations — roughly tied with SemIf's published 0.819 and 0.766. The README states the paired difference on the base set is −0.007 with a confidence interval spanning zero, and explicitly says "we claim no superiority." The 1.7B model is about twice as fast but scores 0.678 on the base set, a paired difference of −0.134 from the 4B — clearly worse, not just slightly.

The known limitations section is unusually candid. The model answers confidently when evidence is missing in 6 of 36 test cases where abstention was correct, compared to SemIf's 1. The maximum of 26 options per question (versus Jev's 255) is a hard architectural constraint from the 26-letter alphabet. And the entire test history comes from one machine: Windows plus an RTX 5060 Ti. macOS, Linux, AMD, and Intel hardware are all untested by the authors, though the Vulkan build was verified to give identical answers to CUDA on the same card.

For agent builders, the practical value is straightforward: structured decisions from a local model, with no output parsing, no JSON repair, and latency around 50 ms per decision at Q8_0 on capable hardware.