All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

The core problem ScriptMoE addresses is not exotic: most deployed OCR systems either run one recognizer per language (cascading error from a language-ID step that, once wrong, cannot be recovered downstream) or lean on massive vision-language models that are too heavy for edge deployment and still collapse on scripts like Bangla, Arabic, and Tibetan. Neither option scales gracefully to the 229 languages the authors target.

The architectural answer is compact. A single hierarchical visual encoder feeds a Transformer decoder whose feed-forward layers are replaced by a sparse mixture-of-experts block. The routing decision is made once per image — not once per output token — by mean-pooling the visual tokens and projecting them through a learned router. That image-level decision selects the top-2 experts from four script-family groups (alphabet, CJK, Arabic, and a catch-all Others), while a fifth shared expert stays permanently active to absorb cross-script knowledge: digits, punctuation, perspective geometry, and the overlapping stroke vocabulary between Chinese and Japanese. The whole model stores about 45.85M parameters; each forward pass activates roughly 41.13M of them.

The data side is equally deliberate. Real scene text data for anything outside English and Chinese is scarce to nonexistent, so the authors synthesize 1M samples per script — 10M total — using a pipeline that balances lexical realism against full character-table coverage, handles right-to-left rendering for Arabic, and oversamples rare characters. Training on this synthetic corpus alone already beats the only prior multilingual synthetic dataset (SynthMLT) by a wide margin, and combining it with real English and Chinese data plus a small real multilingual set yields the best results.

The numbers are specific enough to be useful. On the authors' own TextMuSS-Bench (10,899 real images across ten scripts), ScriptMoE reaches 82.06% average word accuracy, beating the strongest retrained STR baseline by 1.31 points, with the gains concentrated on the hardest scripts: Arabic up nearly 3 points, Thai and Tibetan each up roughly 2 points. The end-to-end test is more striking: swapping only the recognizer inside the PP-OCRv5 pipeline lifts its CC-OCR multilingual F1 from 65.71% to 80.89%, edging past Qwen3.5-9B at 80.73% — a model with roughly a hundred times more parameters.

The ablations are honest about where the design choices matter. Four experts beats both two and ten: two is too coarse to specialize, ten gives each expert too few training samples. The shared expert is the single most important component — removing it costs 0.71 points on average and over 3.6 points on Arabic and Tibetan specifically. Image-level routing is essentially equivalent to token-level routing in accuracy while being cheaper and more interpretable.

Three limitations are stated plainly. A synthetic-to-real domain gap persists. Latin-Cyrillic homoglyph confusion (visually identical glyphs mapping to different characters) is reduced but not eliminated, because real Cyrillic training data barely exists. And the end-to-end pipeline is still capped by the upstream PP-OCRv5 detector, whose multilingual detection quality is uneven — most visibly on Latin scripts under strict word-level evaluation.