A spec-only score turns Transformer architecture search into an exact knapsack problem

Parameter count and FLOPs are the default tools for comparing architectures, but they are blind to structure. Two Transformers with identical parameter budgets but different depth-width tradeoffs, head counts, or FFN ratios score identically under both metrics—yet behave differently when trained. Neural Spectral Capacity (NSC) is a closed-form scalar that fixes this by grounding architectural quality in the singular-value spectrum of each weight matrix, treated as a linear Gaussian channel.

The key move is connecting per-matrix mutual information to the Marchenko-Pastur law. Under standard random initialization, the expected singular-value distribution of any weight matrix converges to a deterministic form that depends only on the matrix's dimensions and initialization variance. This means NSC is computable from the architectural specification alone—no model instantiation, no forward pass, no data. At the smallest hidden dimension across the paper's benchmarks, the closed-form approximation already sits below 1% relative error against the SVD-based value.

The layer-wise additive structure is what makes NSC practically useful beyond ranking. Because the score decomposes as a sum of per-layer contributions, and because typical deployment constraints (parameter count, FLOPs) decompose the same way, maximizing NSC under a resource budget reduces to a bounded knapsack problem solvable by exact dynamic programming. The resulting solver, NSC-DP, returns the globally optimal architecture under the proxy—not just the best one visited by heuristic search. On Transformer-XL, it finds an architecture beating the human-designed baseline in seconds on a single CPU core, versus hundreds of seconds for evolutionary search over competing proxies.

The LLaMA-7B pruning result is the sharpest demonstration of where NSC's structural sensitivity matters. When pruning to a 5.7B parameter target via the LoNAS supernet, all candidate subnets share the same depth and attention configuration—parameter count is constant by construction and provides zero ranking signal. NSC scores each candidate by the spectral content of its retained FFN dimensions and completes the search in under half a second on a CPU thread, roughly 5,900 times faster than the strongest baseline (W-PCA at 45 minutes), while producing the highest average accuracy across eight commonsense reasoning tasks without any calibration data.

A consistency ablation on the same search space compares the closed-form score against one computed from the actual trained singular spectrum. The two induce identical Pareto orderings (Kendall and Spearman both equal 1.0), select identical LoRA-rank profiles at every operating point, and yield mean SVD-oracle regret below 0.1%. The architectural shape dominates the trained spectrum by at least an order of magnitude in this space.

One structural consequence worth noting: because NSC is strictly concave in FFN width at fixed other dimensions, Jensen's inequality forces the proxy-optimal per-layer FFN allocation to be uniform under an additive parameter budget. NSC-DP's discovered Transformer-XL architecture has exactly this property—and the paper proves it must.