Hunyuan-A13B Technical Report

Hunyuan-A13B activates 13 billion of its 80 billion total parameters during inference — a sparse MoE design that lets it compete with much larger dense models at a fraction of the compute cost. That core trade-off drives every architectural and training decision in this report.

The pretraining corpus reaches 20 trillion tokens, with particular attention paid to STEM material: the team extracted 250 billion tokens of high-quality STEM text through enhanced acquisition and cleaning processes, then incorporated a multi-dimensional difficulty grading framework to filter what actually enters training. The context window expands in stages, ending at 256K tokens via NTK-aware positional encoding with alpha values of 50 and 1000 for the 32K and 256K phases respectively.

Post-training runs four sequential stages: reasoning-oriented SFT, reasoning-oriented RL, general SFT, and general RL. The RL work uses GRPO with a 150K-sample dataset balanced across mathematics, coding, logic, and science at a 2:2:1:1 ratio. Notably, the KL divergence constraint is removed, allowing more flexible policy updates. Two context-length phases — 24K then 32K — progressively deepen reasoning during RL training.

The dual-mode CoT mechanism is the most practically interesting piece. A /think tag triggers slow-thinking with explicit step-by-step reasoning inside a dedicated block; /no_think produces fast-thinking with an empty think block. Both modes are trained simultaneously in a unified structure, so the model doesn't need separate weights for each. Default behavior, when no tag is provided, falls to slow-thinking.

On benchmarks, the model leads on AIME2024 and ZebraLogic among the comparison set, and scores 87.7 on PenguinScrolls — essentially matching Gemini 2.5 Pro's 88.3 while outpacing DeepSeek R1 and Qwen3-A22B. Long-context stability is a genuine strength: at 64K–128K context, Hunyuan-A13B holds 73.9 on RULER's QA task versus 65.6 for DeepSeek R1 and 66.6 for Qwen3-A22B. On LongBench-v2 it scores 55.0, second only to Gemini 2.5 Pro at 60.9. RAG-specific tasks via FRAMES show a gap — 81.1 against DeepSeek R1's 85.7 — which the report acknowledges directly.

For agent use specifically, the report claims Hunyuan-A13B outperforms larger alternatives on BFCL-v3 and related tool-calling benchmarks. The agent training data includes a five-role synthetic engine (user, planner, tool, agent, checker) and over 20,000 format combinations across tool, action, and response variations — a deliberate effort to generalize across calling strategies rather than overfit to a narrow set of tool schemas.

Inference is supported out of the box with vLLM, SGLang, and TensorRT-LLM, with lossless quantization options including Weight Only INT8, W8A8, and KV Cache FP8. The combination of sparse activation and quantization support makes this genuinely deployable on constrained hardware, not just theoretically efficient.