Runtime abliteration beats weight editing when your model is ternary-quantized

Abliteration — the technique of removing a model's refusal behavior by orthogonalizing its weight matrices against a learned refusal direction — breaks down when the model is aggressively quantized. You cannot save the edited matrix back into a ternary representation without re-quantizing, and re-quantizing undoes the quantization-aware training that made the model good in the first place. This repo solves that by never touching the weights at all.

Instead, the projection happens at inference time. After each residual write, the runtime subtracts the component of the activation that lies along the refusal direction: y ← y - α · dot(y, r) · r. The original ternary codes stay bit-identical. The behavioral change is a runtime parameter.

The architecture of Ternary Bonsai 2 makes this more involved than wrapping a single attention output projection. The model has 48 linear-attention layers, 16 full-attention layers, and 64 MLP blocks — all of which write into the residual stream. Wrapping only o_proj would intercept 16 of the 129 sites that actually matter. The repo wraps all of them.

The evaluation numbers are honest about their limits. Refusal is judged by a rule-based phrase classifier, not an LLM judge. The SimpleSafetyTests result is explicitly flagged as understated: the model responds to self-harm prompts with crisis redirects that the classifier misses, scoring them as compliance. On AdvBench, refusal drops from 99% to 6% at alpha=1, with 56% of responses still wrapped in disclaimers. The over-refusal story is more striking — the base model refuses a quarter of JailbreakBench's benign prompts; ablated, it refuses none.

Capability retention across MMLU, GSM8K, and CMMLU shows movements within noise at the sample sizes used. That is the expected result when weights are untouched.

The repo also ships a GGUF LoRA adapter for llama.cpp users, though it requires PrismML's fork — stock llama.cpp rejects the private ternary quantization types outright. The adapter is rank-1 by construction, since the weight edit W - r(rᵀW) is rank-1, and it was only tested against the PTQ1_0 pack.

One caveat the repo states plainly: the refusal direction was estimated from the BF16 base model, not from the QAT model itself. Whether it transfers cleanly through quantization-aware training is not fully measured. The runtime can verify it is removing the supplied direction from the residual stream; it cannot verify that direction means the same thing in the QAT model.