🔍 Read the full analysis: What Happens When GPT-6 Astra Tunes Another AI Model’s Weights? on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A November 2 blog post by four researchers listed with MIT CSAIL reports that GPT-6 Astra modified two layers of Qwen3.5-0.8B, raising its MATH-500 score from 7.9% to 48.1%. The edit used iterative testing rather than gradient descent, and the model still hit its answer-length limit on 33.1% of benchmark problems. The work is an early, non-peer-reviewed result, not evidence that the model gained general mathematical ability.
Four researchers listed with MIT CSAIL say OpenAI’s GPT-6 Astra improved the math benchmark performance of the open-weight model Qwen3.5-0.8B by editing a small part of its weights, without using gradient descent. In their November 2 blog post, the researchers report that the model’s MATH-500 score rose from 7.9% to 48.1%, while answers reaching the token limit fell from 93.0% to 33.1%. The result describes a targeted repair of repetitive outputs, not proof that Astra taught the model mathematics or removed the problem entirely.
The post, titled Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights, identifies Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab as authors; Takehi and Teoh are marked as equal contributors. The authors describe the work as “very early results.” It is a blog post, not a peer-reviewed paper, and the source says it was not on arXiv. OpenAI is not listed as an author, and the post does not report a statement from the company about the experiment.
According to the researchers’ published logs, Astra used a Python-based research setup to inspect and edit the 24-layer Qwen model. It identified a direction associated with looping in the multilayer perceptrons of layers 15 and 19, then subtracted that direction from 64 output columns in each layer. The change affected 131,072 weights, which the authors say is 0.016% of the model’s 0.8 billion weights. The edited Hugging Face model, according to the source, differs from the original in the two layers’ down-projection weight tensors.
The improvement depended on an iterative process, not a one-shot instruction. In the reported 50-pass run, Astra used 158 turns, 80 Python runs and 50 forward passes over 20.3 hours. Each forward pass could evaluate up to 512 problems. The researchers say Astra was given practice problems but not the final benchmark sets. The MATH-500 scores were 12.3% after 10 passes and 14.1% after 25, compared with 48.1% after 50. At the same time, the share of answers reaching the 32,768-token limit remained 87.2%, 86.4% and 33.1%, respectively.
A Targeted Fix, Not New Math Skills
The result matters because it suggests that a capable model may be able to help diagnose and alter behavior inside an open-weight model without the usual gradient-descent training process. For teams that can inspect and modify model weights, this could offer another way to address a specific failure mode. The reported edit was small relative to the model, but it still required substantial testing, access to the weights and a carefully configured evaluation setup.
The benchmark gains also need to be read alongside the remaining failure rate. After the edit, 33.1% of MATH-500 answers still reached the token limit, so the looping behavior was reduced rather than eliminated. The authors themselves caution against equating that repair with a general improvement in mathematical reasoning. The result is evidence about one model, one identified behavior and one experimental process; it does not establish that the method will work reliably across models or tasks.
There is also a governance implication for organizations using open weights. A modified model is no longer identical to the published base model, even if only a small number of weights change. It needs its own testing and documentation, particularly if a patch improves one measured behavior while affecting others. The source notes that related single-direction edits have been studied as a way to remove refusal behavior, underscoring that targeted edits can carry consequences beyond the intended repair.
Products Worth Considering
Apple MacBook Pro 14.2-inch with M5 Max, 18-core CPU, 32-core GPU, 36GB Memory, 2TB SSD, Wi-Fi 7, Silver
Apple MacBook Pro 16.2-inch with M5 Pro, 18-Core CPU, 20-Core GPU, 48GB Memory, 1TB SSD, Wi-Fi 7, Silver
Apple MacBook Pro 14-inch with M5 Max Chip, 18-Core CPU, 20-Core GPU, 48GB Memory, 2TB SSD, Space Black
As an affiliate, we earn on qualifying purchases.
How the Weight Edit Was Tested
Qwen3.5-0.8B is a small open-weight model. The source says its model card warns that it is prone to thinking loops. On the researchers’ initial practice problems, 85 of 96 answers reportedly ran to the model’s 32,768-token limit. That failure gives context to the benchmark result: the experiment was aimed at preventing the model from repeating itself until it ran out of room, not simply maximizing a score through additional math training.
The reported evaluation included MATH-500, GSM8K and IFEval. On GSM8K, the score rose from 13.8% for the base model to 67.1% after the 50-pass run; on IFEval, it rose from 21.0% to 62.3%. These figures are reported in the blog post and should be understood as results from its stated setup, not independent replication. The source also describes other experiments in the same post, including a failed attempt to identify which of two fine-tuned Gemma models had been trained to flatter users.
The researchers distinguish their method from conventional training: they say gradient descent, backward passes and fitting weights to a loss were prohibited. But “without training” does not mean without computation or feedback. Astra repeatedly examined model outputs, ran code and tested candidate edits. The post reports that smaller budgets had much weaker results, and that a no-loop model did not improve in a comparable test. The amount of evaluation was therefore central to the reported outcome.
““very early results””
— Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, in their blog post
Replication and Side Effects Remain Open
The reported scores have not been established through peer review or an independent replication in the material provided. The post supplies logs and the edited model, but the source does not describe outside researchers reproducing the full result. It is also unclear whether the same weight-editing approach would work on other model sizes, architectures or looping behaviors.
The reported data do not settle whether the edit affects performance outside the listed tests, or whether reducing repetitive outputs introduces other weaknesses. The source says the benchmark sets were withheld from Astra, but does not provide evidence here about performance across a wider range of tasks or deployment conditions. The remaining token-limit rate shows that the targeted behavior persisted for a substantial share of MATH-500 answers.
The source also reports that a separate attempt to infer a model’s training objective from its weights failed. That result is a reminder that weight inspection did not succeed across every experiment in the post. OpenAI’s position on the Astra run is not included, and no claim of company participation or endorsement is established by the authors’ use of the model.
Independent Tests Will Matter
The immediate next step for evaluating the claim is independent testing of the published edit and logs, including checks on the same benchmarks and on tasks beyond those reported. Further work would need to establish whether the loop reduction persists across prompts, sampling settings and other model versions, and whether the weight change causes regressions elsewhere.
Readers should also look for any follow-up from the researchers, publication in a peer-reviewed venue, or a response from OpenAI. Until then, the available evidence is the authors’ blog post, their published logs and the edited model identified in the source. The current conclusion is limited: the team reports a measurable improvement in a specific setup, with substantial residual looping and unresolved questions about generality and side effects.
Key Questions
What did GPT-6 Astra change?
According to the researchers, Astra identified a direction associated with looping and subtracted it from output columns in the model’s MLP layers 15 and 19. The edit changed 131,072 weights, which the post says is 0.016% of Qwen3.5-0.8B’s weights.
Did the method use no training at all?
The researchers say they did not use gradient descent, backward passes or loss-based weight fitting. However, Astra used iterative output checks, Python runs and forward passes to assess candidate edits. “Without gradient descent” is more precise than saying no work or evaluation was involved.
Did the edit eliminate the model’s loops?
No. The reported share of MATH-500 answers reaching the 32,768-token limit fell from 93.0% to 33.1% after the 50-pass run, but a substantial portion still hit the limit.
Has the result been peer-reviewed?
The source describes the work as a blog post published November 2, 2026, and says it is not peer-reviewed or on arXiv. The authors call the findings “very early results.”
Does the result show that Qwen learned mathematics?
Not by itself. The reported experiment targets repetitive outputs that interfered with answering problems. The researchers explicitly caution that fixing loops is not the same as making a model better at math.
Source: ThorstenMeyerAI.com
