Runtime ablation keeps ternary weights bit-identical where weight editing cannot

Refusal-direction ablation on quantized models has a genuine technical problem: the standard weight-editing formula produces a dense float matrix, and saving that back into a ternary representation requires re-quantization. Re-quantization without the original quantization-aware training process degrades the model in ways that are hard to characterize. This repo sidesteps the problem entirely by moving the projection to inference time.

The math is simple. Instead of modifying weight matrices, every residual write gets intercepted and the component parallel to a learned refusal direction is subtracted before it enters the residual stream. The original ternary weights — approximately 1.72 bits per weight across 27 billion parameters — are never touched. A single scalar alpha controls how aggressively the direction is removed, and it can be set to zero to restore the original model behavior from the same checkpoint.

The coverage is more thorough than most abliteration implementations. Ternary Bonsai 2 uses a hybrid architecture with 48 linear-attention layers, 16 full-attention layers, and 64 MLP blocks, each capable of writing into the residual stream. The runtime wraps all 129 of those sites. Wrapping only the standard attention output projection would catch 16.

The evaluation numbers are honest about their limits. The refusal classifier is rule-based, not an LLM judge, and the README flags a specific miscounting problem on SimpleSafetyTests where crisis-redirect responses get scored as compliance. Capability benchmarks — MMLU, GSM8K, CMMLU — show movements well within noise at the sample sizes used, which the README attributes directly to the bit-identical weights. The over-refusal results are striking: the base pack refuses a quarter of JailbreakBench's benign prompts; the ablated version refuses none.

There's also a GGUF path via a rank-1 LoRA adapter that keeps the base GGUF byte-identical, and a Swift reference implementation for iOS. The iOS section is unusually candid about memory constraints — the full pack sits at roughly 8 GiB, and the README explicitly warns that an 8 GB device cannot fit 6.756 GiB of weights plus an app.

The direction transfer caveat deserves attention. The refusal direction was estimated from the BF16 base model, not from the QAT-trained ternary version. The runtime can verify mathematically that it is removing the supplied direction from the residual stream. Whether that direction captures the same behavioral feature in the quantized model is a separate question, and the README says it has not been fully measured.