The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning

Popular facts resist erasure because they are encoded more deeply during pretraining, not because they are harder to suppress at the surface. That distinction drives everything in this paper. Existing gradient-based unlearning methods apply the same pressure to every fact regardless of how often it appeared in training data, which produces a predictable split: rare facts get over-erased, damaging retained knowledge, while popular ones survive under paraphrase or adversarial prompting even when their direct recall looks low.

AdaPop addresses this by pulling the popularity signal from outside the model entirely. A Wikidata sitelink count - or an LLM-as-Judge score when Wikidata coverage is absent - is converted into a per-fact power-law exponent that shapes how each token's gradient behaves as training proceeds. The math is deliberate: for shallowly encoded facts the exponent sits above a crossover value where the gradient attenuates as tokens are forgotten, naturally halting erasure before it damages adjacent knowledge. For popular facts the exponent sits below that crossover, sustaining gradient pressure even as confidence on those tokens drops. A dual-ascent controller then adjusts the retain penalty each epoch rather than requiring a per-dataset grid search.

The internal representation metrics are the most telling part of the evaluation. WGA, the closest competitor, achieves low surface ROUGE-L on forget sets but shows a negative Rank shift - meaning the gold token actually moves up in the ranked distribution after unlearning even as its absolute log-probability falls. That is a signature of output flattening rather than genuine erasure, and it explains why WGA's forgotten facts remain recoverable under adversarial reformulations. AdaPop's forget-set hidden states move further from their pre-unlearning positions than any stable competing method, while retain-set representations stay close.

The proxy ablation is reassuring about practical deployment. Wikidata scores, LLM judgments, and direct corpus-frequency counts from Pile-train all reach comparable forget quality by the primary reported learning rate. The Wikidata score is more stable at higher rates because its anchor calibration was derived from that distribution; the LLM judge loses up to several points of retention at higher rates while achieving no additional forgetting. Corrupting scores with label noise or even inverting every rare/popular label degrades retain by only a few points, because the dual-ascent controller raises the retain coefficient when it detects drift.

The stated limitations are real. Wikidata coverage ends at entity-centric factual knowledge; procedural or creative content has no equivalent proxy, and the authors acknowledge that the LLM judge is a fallback rather than a replacement. All experiments use LoRA fine-tuning at the 7-8B scale; full fine-tuning is excluded. The exponent coefficients were derived from the DUET score distribution and need recalibration for proxies with fundamentally different dynamic ranges.