IntBMoE solves MoE's three-way tradeoff and proves it in production at AMap

Every MoE design so far has been forced to trade one of three quantities against the others. Sparse routing keeps compute low but means most experts never touch a given token. Dense output mixing lets every expert contribute but bills you for running all of them. Parameter merging collapses the pool into one composite expert but must rebuild that composite for every distinct routing decision, so memory grows with traffic. IntBMoE breaks all three couplings at once.

The mechanism is a two-stage separation. First, a small learned codebook of block embeddings drives a shared hypernetwork that composes the full expert pool, layer by layer, into a finite set of reusable multi-layer blocks. Because the codebook is input-independent, these blocks can be precomputed and cached before any request arrives. Second, a lightweight router sends each token to only a few of those cached blocks. Participation is full because every block draws on every expert basis. Execution is sparse because only the selected blocks run. Materialization is bounded because the codebook size, not the number of tokens or routing decisions, determines how many composed parameter sets exist.

Within each block, Dual-Path Residual Gating composes the expert bases twice—once into a value path and once into a gate path—then multiplies them residually. That multiplicative interaction is nonlinear in the expert bases, adding expressiveness without enlarging the pool or the codebook.

On ImageNet-1K with an eight-layer DeiT-Tiny-style backbone, IntBMoE reaches 73.76% Top-1 accuracy, a 7.36 percentage-point gain over the dense baseline and 1.98 points above the next-best competitor, SMEAR. The ablation is unusually informative: removing the gate path causes the largest single drop, collapsing each two-layer block to a parameter-matched single layer is the second-largest, and even fixing the residual scale or softmax-normalizing the composition coefficients both hurt. Every expert basis matters too—removing any one of the 16 bases examined across four layers reduces accuracy, with the smallest observed drop at 0.26 percentage points and the largest at 9.15.

The architecture generalizes. On MiniPile language modeling it cuts perplexity by 12.4% relative to the dense backbone and by 2.9% relative to the strongest MoE baseline. On the IntTravel sequential recommendation benchmark it leads on all three reported metrics.

The production story is the sharpest validation. AMap deployed cached IntBMoE in its map-screen POI recommender, handling roughly 5,000 queries per second on Alibaba T-Head PPUs. A one-week A/B test showed a 2.4% relative UVCTR gain while staying within the latency budget. The caching insight is what makes that possible: with parameters fixed at inference time, block synthesis is a one-time cost, and request-time compute is independent of expert-pool size.