Astra edited 0.016% of Qwen's weights and lifted MATH-500 from 7.9% to 48.1%

Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL have introduced Weight-Space Programming. A frontier model receives access to the tensors of a smaller network and must identify, repair or create behavior from a natural language goal by changing weights directly, without gradient descent or a conventional training loop.

Astra traced a reasoning loop and changed 131,072 weights

The main experiment used Astra on Qwen3.5-0.8B. The model often repeated its reasoning until the 32,768 token limit. Astra traced the problem to one direction in the MLPs of layers 15 and 19, then modified 64 output columns in each. It changed 131,072 weights in total, about 0.016% of 0.8 billion.

With 50 forward passes, MATH-500 rose from 7.9% to 48.1%, GSM8K from 13.8% to 67.1% and IFEval from 21.0% to 62.3%. Answers hitting the token limit fell from 93.0% to 33.1%. The authors published a technical blog and experiment logs, not a peer reviewed paper.

Direct weight repair could shorten the path from diagnosis to intervention

A conventional workflow separates interpretation, data preparation, training and evals. Here one model analyzes tensors, runs bounded tests and proposes a small intervention. If reproduced, the approach could help repair specific failures where the defect is known and collateral damage can be measured.

Other demonstrations include identifying simple policies from weights, repairing a control policy by changing 3.5% of its parameters and generating small game playing networks. The practical value therefore sits with small, observable models today rather than autonomous reconstruction of frontier systems.

The large score gain mostly removed a pathological loop

The 7.9% MATH-500 baseline was unusually low because 93% of answers ended at the token limit. The edit primarily removed an obstacle that prevented completion. It does not show Astra writing a new mathematical capability into Qwen.

Direct inspection of dense internal representations also failed repeatedly in one experiment until the model used activation probes. The method sits between mechanistic interpretability, automated intervention search and weight programming. Its name currently covers a broader ambition than the demonstrated scope.

Independent replications must measure transfer and collateral damage

The next step is reproduction across architectures and failure modes with predefined evals. The important questions are whether the intervention works without a pathological loop, holds outside calibration data and preserves capabilities that already worked.

Production use would require an exact weight diff, rollback, an audit of every test and comparison with cheaper decoding changes. If a sampling adjustment solves the same failure, a weight space operation may be impressive but unnecessary.

Lilith's verdict

Astra did not pour new mathematics into Qwen. It found a jammed gear, filed 131,072 teeth and let the machine reach an answer. That more sober version is the technically interesting one.