Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization

Retrieval-free question answering over a fixed document corpus is harder than it looks. Standard supervised fine-tuning on QA pairs exposes the model only to the facts the question generator happened to select. Continued pretraining on raw documents gives denser exposure but doesn't teach the model to answer questions. Both routes tend to erode general instruction-following ability—a model that can't follow ordinary instructions isn't deployable, regardless of how well it internalized a corpus.

IAR addresses this by splitting adaptation into three explicit stages. Inject converts documents into continuation, rewrite, and instruction-conditioned reconstruction tasks—supervised objectives with richer document coverage than QA-only training, but without the raw language-modeling loss of continued pretraining. Align then fine-tunes the injected checkpoint on answer-only QA supervision. Recover applies post-hoc weight-space merging between the domain-adapted model and the original instruction model, selecting an operating point that balances domain accuracy against general capability.

The empirical case is methodical. Against vanilla SFT, IAR improves on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 points in mean general performance across IFEval, MMLU, and MSBench. A token-budget control—BudgetMatch, which runs QA-only training for enough epochs to match the Inject+Align token volume—shows that simply training longer on QA pairs explains part of the CC gains but not the CCI results, and IAR beats BudgetMatch on domain accuracy in three of four settings while winning all four mean-general comparisons.

The Qwen3 scaling results (8B, 14B, 32B on CC) show Recover trading at most 1.1 domain points off the pre-recovery peak while recovering 14.9 to 24.1 points in mean general performance. The Inject+Align contribution doesn't disappear at scale: relative to vanilla SFT, the pre-recovery checkpoint adds 8.8, 5.7, and 7.5 domain points at those three sizes, and IAR retains most of that after merging.

The paper is honest about its limits. Phi on CCI is a genuine boundary case where IAR improves IFEval and MSBench but slightly reduces domain accuracy and MMLU relative to vanilla SFT. Qwen3-4B on CCI starts from an unusually high 70.6% base domain score, leaving little headroom; a bits-per-byte diagnostic suggests Qwen3 fits the CCI text distribution better than the other model families before any training, which complicates interpretation. No single Inject recipe is universally best across corpora and model families. And the scaling runs are missing their original training logs, so the computing-environment description doesn't cover those three rows.

What the framework actually contributes is a controlled decomposition. By separating document exposure, QA alignment, and capability recovery into auditable stages—each with its own ablation—it makes visible which intervention explains a gain or failure, rather than collapsing everything into one fine-tuning comparison.