Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification

Weights remember where they came from. That is the core claim here, and the mechanism is specific enough to be useful: when a residual network trains, its input and output projection matrices within each block develop a structural relationship that persists through fine-tuning, quantization, pruning, and LoRA merging. The branch product of those projections concentrates energy along the identity direction - a property that appears across six language model families tested, with block-pairing accuracy hitting 100% on the canonical MLP path against random-initialization baselines of at most 4%.

The catch is that this identity-aligned structure is generic. Any two independently trained residual models share it, so it cannot establish ancestry on its own. The paper's contribution is the centering step: strip out the identity component, and what remains is checkpoint-specific. Compare those centered residuals across two models and you get a lineage score that sits near 1 for descendants and near 0 for independently trained models. On both a controlled MLP benchmark and a 30M-parameter GPT-2 benchmark, the score achieves AUROC of 1.0, cleanly separating fine-tuned, pruned, quantized, and LoRA-merged descendants from distilled students and independent models.

The robustness story is where this gets interesting. Simple weight-space baselines - cosine similarity, Frobenius distance - also achieve perfect discrimination on clean benchmarks. But shuffle the hidden units of a derived checkpoint (a function-preserving operation that changes nothing about what the model computes), and weight cosine's Gap-Z score collapses from strong separation to near-overlap, even when AUROC stays nominally above 0.8. The centered residual signature is algebraically invariant to exactly these permutations and reciprocal rescalings, so its margin holds. The one competing method that also survives laundering, Re-Basin+scale, does so by solving a Hungarian assignment problem per layer pair - which costs 76 times more latency on GPT-2 than the branch-product approach.

A public LLaMA-2 case study correctly scores three documented derivatives high and all seven independently trained architectural clones near zero, including OpenLLaMA, Amber, Baichuan, InternLM, and Yi - models that share identical architecture but were initialized from scratch.

The limitations are stated plainly. The method requires white-box access to both checkpoints and matching depth and hidden dimension. It cannot determine which model is the ancestor. Heavy pruning and extensive continued pretraining degrade the signal, though both remain detectable above the null in tested conditions. Orthogonal rotation of the residual stream would defeat the signature, but for standard LayerNorm and RMSNorm architectures this rotation is not function-preserving, which limits that attack vector in practice. SwiGLU architectures reach only 80% block-pairing accuracy and need joint factorization to recover sub-paths reliably.

For anyone auditing open-weight model supply chains - tracking whether a released checkpoint descends from a base model it claims no relation to - this is a practical, data-free tool that works without any proactive watermarking.