🔍 Read the full analysis: 5 Key Findings From Our Home Weight-Space AI Editing Test on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment used directional ablation to reduce looping in the local AI model Ornith 1.5 35B-A3B. On the same setup, it completed 10 of 10 published Lexassay tasks, up from 4, but its legal score remained zero and its answers contained 55 wrong or invented citations. The result suggests the edit improved completion, not legal knowledge; the test was small and had significant limitations.
A home experiment reported by ThorstenMeyerAI.com found that a weight edit reduced looping in the local model Ornith 1.5 35B-A3B, helping it complete 10 of 10 published legal benchmark tasks, compared with 4 of 10 on the same setup without the edit. Its legal score nevertheless stayed at zero, underscoring that finishing an answer is not the same as giving a correct one.
The experiment ran from Wednesday evening to Thursday morning and was inspired by an MIT CSAIL study of direct edits to neural-network weights. The home test used a related but different method: a Claude Opus 5.5 session wrote code to run Ornith, detect a looping direction in its internal signals, and remove that direction at half strength from the output weights of 27 of its 40 layers. The team says it did not train the model, add legal data, or have an AI read and edit the weights directly as text.
The fair comparison used the same setup and the 10 published Lexassay tasks. The unedited model answered four; the edited model answered all ten. Yet the edited answers met only 7 of 50 checked criteria, up from three, and included 55 wrong or invented citations. The benchmark checks cited statutes and court decisions against primary sources. Lexassay also has five hidden tasks, but the figures in this report cover only the published ten.
The report describes another result that complicates the comparison: with sampling settings recommended by the model card, the unedited model answered 9 of 10 tasks on its original setup. In a separate published run it answered eight of ten. Those runs used different local setups and are not directly comparable with the four-of-ten baseline used to measure the edit. An independent reviewer, OpenAI’s GPT-6 Astra running in Codex, checked the code, figures and claims and found faults, according to the report.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Fewer Loops, No Legal Gain
The result draws a practical distinction between reliability of completion and subject-matter capability. An edit can apparently help a model stop repeating itself and reach the end of a response. In this test, that improvement did not translate into accurate legal analysis: the score stayed at zero, while the answers contained dozens of citation errors.
That matters for people evaluating local AI systems for professional or research tasks. A response that arrives on time may look more usable, but completion alone does not establish that its reasoning, facts or sources are sound. The report’s findings do not show that weight editing improves legal performance, nor do they establish how the technique would perform on other models or larger tests. This is an experiment, not legal advice.
Products Worth Considering
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Apple MacBook Pro with 16.3-Inch PROLED XDR Display (2026, Silver)
LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz
As an affiliate, we earn on qualifying purchases.
From MIT’s Weight-Editing Study
The home test followed an MIT CSAIL report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab on what they call weight-space programming: changing a model’s weights without a conventional training loop. In the MIT team’s Qwen example, GPT-6 Astra identified a direction associated with looping and subtracted it from a small share of the model’s weights; the report said the model’s mathematics score rose from 7.9% to 48.1%.
The home experiment did not reproduce that procedure. Instead, it used directional ablation, a technique that measures a direction in a model’s activations and removes it from weights. The team selected Ornith after checking four local models: it was the one that showed the relevant looping behavior and could be tested on the available machine. The report says the other candidates either failed for reasons other than looping or could not be run under comparable conditions.
Lexassay tests models on questions involving German, U.S. and European Union law, without internet access or tools. Its published tasks and checked answers are available on the benchmark site, according to the report. Ornith’s publisher describes the model as designed for coding and agent work, not legal analysis.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, reporting on the home experiment
Limits of the Ten-Task Test
The results are based on one run per setting and ten published tasks, not repeated trials across a large or representative legal benchmark. The report also says the loop detector changed during the experiment, and that three graders came from the same model family. GPT-6 Astra reviewed the work and identified faults, but that review was not a second, competing experiment.
The different completion rates for the unedited model—four, eight and nine tasks across separate runs—cannot be treated as a single direct comparison because the local setups and sampling settings differed. It is also unclear from the reported findings whether the edit would hold up across repeated runs, other models, the five hidden Lexassay tasks or legal questions outside the benchmark. The evidence supports a narrow claim about task completion under one test setup, not a general claim of improved legal ability.
Public Runs and Further Testing
The report says the experiment’s answers, grades and citation checks are public on Lexassay, alongside the method description. Those materials allow readers to inspect the unedited run, model-card sampling run, edited run and edit control rather than relying only on the summary figures.
The next useful evidence would come from repeated, controlled runs using the same hardware, token limits and sampling settings, followed by tests on the benchmark’s hidden tasks and other models. The report does not announce a scheduled follow-up or claim that those tests have been completed. Until there is broader evidence, the finding remains limited: the edit helped this model finish more published tasks on one setup, while its measured legal performance remained poor.
Key Questions
What did the weight edit change?
Code written during a Claude Opus 5.5 session measured a looping direction in Ornith 1.5 35B-A3B and removed it at half strength from the output weights of 27 of 40 layers. The report says the model was not retrained and no legal data was added.
Did the edited model become better at law?
Not according to the reported score. It completed more tasks on the same test setup, but its legal score stayed at zero. The edited answers met seven of 50 checked criteria and contained 55 wrong or invented citations.
Why do some results say the unedited model answered more tasks?
The report gives results of four, eight and nine completed tasks for the unedited model across different runs. It says those runs used different local setups or sampling settings, so they are not directly comparable. The edit’s 4-to-10 comparison is the reported fair comparison on the same setup.
How strong is the evidence?
It is a small experiment: one run per setting, ten published tasks, a loop detector that changed during testing and graders from the same model family. The report says an independent GPT-6 Astra review found faults. The results do not establish that the technique will work on other models or improve performance on legal tasks generally.
Can readers inspect the test?
According to the report, the runs, answers, grades and citation checks are public on Lexassay.com, with a separate method page describing the experiment. The findings concern an AI benchmark experiment and are not legal advice.
Source: ThorstenMeyerAI.com
