The 4-bit model that forgot its worse state and plays in a better league

The original architecture as the only measure of success

The standard procedure for shrinking LLMs (architecture reduction followed by quantization) usually comes with a hefty price in lost capabilities like reasoning and coding. That is why models are healed using Quantization-Aware Training (QAT). However, Hugging Face and Multiverse Computing are introducing Quantization-Aware Healing (QAH), where a 60B parameter MXFP4 model distills knowledge directly from the original large 120B teacher, not from a trimmed-down version.

Who gets smaller demands without compromises

The new 4-bit models need 4x less memory and half the compute, yet the QAH model beats its own full 16-bit (bfloat16) version in 7 out of 9 benchmarks. Instead of the quantized model learning with errors (as cross-entropy QAT does), it captures knowledge from precise logits using KL divergence. This will significantly relieve teams needing to deploy smaller, cheaper, locally running agents.

When distillation starts limping in extreme tests

The reality check is present, however. In extreme long-context scenarios (like the AA-LCR benchmark), the 4-bit QAH model still trails the giant 120B teacher because raw architectural capacity cannot be substituted. It also needs to be considered that this is currently a paper and a demo; real-world adoption outside Hugging Face models will become apparent over time.

Speed and stability will decide

QAH reaches its peak roughly 7x faster (in 100 steps) than competing QAT and, crucially, does not collapse. While the QAT model sharply degrades after 1,200 steps, QAH remains stable. The key signal will be whether this circumvention of the 'degraded intermediary' is adopted by other creators of smaller models.

Lilith's verdict

Skip the bad compression phase, and you get an agent that doesn't need to crack unnecessary hallucinations. This grants local teams full power over a model that previously only ran in the cloud.