Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Accuracy on a translated benchmark is nearly useless for evaluating a low-resource reasoning fine-tune. That is the paper's first and most important finding, and it arrives with a precise measurement: rerunning the same configuration with a different random seed moves the score by 7.7 points, more than any data choice, corpus version, or recipe change the authors tested across the entire project. Every effect they had been interpreting was smaller than their ruler's precision.

What actually changed under fine-tuning lives in dimensions accuracy cannot reach. Base models never reason in Greek — zero of a thousand traces cross the threshold for Greek script, even when the question is Greek. The model answers correctly while thinking in a language its user cannot read or audit. After supervised fine-tuning on language-matched pairs, every released checkpoint reasons in the question's language on 98% of items, with judged grammaticality improving across all four model families and general ability within a few points of each base in both languages.

The token cost of this shift is family-dependent in sign. Greek costs more tokens per word than English on all three families — a fertility gap the paper measures per tokenizer and treats as the hidden hand behind every cost comparison. The Qwen release shortens its traces enough to come out ahead in tokens; the NemotronH fine-tune reaches parity; the Gpt-OSS release ends up more expensive to serve than its English-reasoning base. The word-level reasoning effort drops everywhere; the serving bill does not.

Reinforcement learning with verifiable rewards, pre-registered before any training, fixed two defects SFT left behind: answer-format fallback and answer-channel leak both dropped past their pre-registered thresholds, while a random-reward control trained on identical data and steps reproduced the baseline on every axis — confirming the improvement is learning, not elicitation. A third target, explicit language-override compliance, moved 14 percentage points but missed its pre-registered trainability bar, so the paper calls it reward-responsive rather than solved.

The methodology section is as valuable as the results. Six instrument failures are documented in full: a scorer that ignored the requested answer line and penalised verbose models; a commonsense deficit that reversed entirely when the answer format was constrained; benchmark contamination from drawing training and evaluation data from the same pool; a register artifact from translating test items with the same model family used to generate training traces; a numeric parser that misread Greek thousands separators; and a generation harness default that silently measured the wrong inference mode for hours without raising an error. Each failure had already produced a plausible, written-up result before the control caught it.

The practical upshot is direct: run the seed control before any ablation, report format-compliance rates alongside accuracy, and keep a human-translated subset of any machine-translated benchmark. A fine-tune that leaves accuracy flat is not a failed fine-tune — it may have changed everything that matters for deployment.