🔍 Read the full analysis: I Put 71 AI Model Setups To Work On My News Pipeline on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer says he tested 71 AI model configurations on a production news-writing task from Sept. 8 to Oct. 13, 2026. GPT-6 Astra scored highest, but GPT-6 Luna was selected for production because it performed consistently at about one-hundredth of Astra’s reported cost. The test also found fabricated quotes in two fallback models, which Meyer says were removed.
Publisher Thorsten Meyer says he tested 71 AI model configurations on his news-writing pipeline between Sept. 8 and Oct. 13, 2026, and selected GPT-6 Luna for production despite GPT-6 Astra receiving the highest scores. Meyer reports that Astra scored 4.00 out of 5 in English and German, while Luna cost about one-hundredth as much per article and scored 3.67 in the robustness test.
Meyer’s test used his production prompt and real news sources, with each source turned into an English and a German article. The configurations covered API and subscription models, free-tier models accessed through OpenRouter, and open-weight models run on Meyer’s own hardware. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored outputs on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles also received a language-quality score.
The initial test used two sources: a 6,340-character product filing and a 619-character political news item. Meyer says most stories in his pipeline begin with short sources, making the shorter item more representative of routine use. Stronger-performing configurations then completed a further 10 articles, mostly from short sources. Meyer says results include 95% bootstrap intervals and that small gaps in the two-article initial test should be treated cautiously.
In the standard test, Astra scored 4.00 in both languages and received 5.00 for source fidelity. Across 12 articles in the robustness run, it scored 3.93. Luna scored 3.67 in that run, with source fidelity of 4.71; Meyer says every Luna output was valid for his pipeline. He estimates Astra at about $0.12 per article and Luna at about $0.0012, with typical generation times of roughly 50 seconds and 17 seconds, respectively. In his reported production use over the previous week, Luna cost about $0.0013 per article at a median of 17 seconds.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Why Luna Made the Production Cut
The results highlight a practical distinction for publishers using AI: the highest-scoring model may not be the workable production choice. Meyer says Astra’s quality advantage did not justify a cost roughly 100 times higher for his high-volume pipeline, while Luna cleared his quality threshold at much lower reported cost. That is a decision specific to his workflow, sources, prompt and scoring method, not a general ranking for every newsroom.
The fallback findings carry a separate operational lesson. Meyer reports that one backup model generated invented quotations and another failed to produce any valid articles in four attempts. A pipeline’s output depends not only on its preferred model but also on what happens when that model is unavailable. Testing failover models for factual fidelity and valid formatting can matter as much as comparing headline scores.
Meyer also found no reliable benefit from increasing Luna’s reasoning setting. In his test, higher settings added time and cost without a meaningful score improvement. This suggests that, for this particular source-to-article task, more computation did not reliably improve source discipline; it does not establish that reasoning effort is unhelpful for other tasks.
As an affiliate, we earn on qualifying purchases.
How the Pipeline Test Worked
The report addresses a gap Meyer sees in public model leaderboards: broad benchmark scores do not directly identify which model will perform best on a specific newsroom’s daily writing task. His evaluation was designed around one defined job: turning supplied news sources into finished articles using the prompt his own pipeline uses.
To judge whether outputs could be used operationally, Meyer counted an article only if it produced valid output after the same repair step used in production. He also says he used two judges that both saw the original source. Because the initial comparison involved only two source articles, Meyer cautions that score gaps below about 0.3 points could be noise and places more weight on the additional robustness run.
The report compares English and German editions and includes models with different access methods and operating costs. Its results therefore reflect Meyer’s task, evaluation criteria and pipeline constraints. The provided material does not describe an independent audit or a human-only control group.
“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in the report
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Scores
The findings come from one publisher’s pipeline and a limited set of source material. The standard comparison began with two articles, and the larger robustness run covered 12 articles for the reported Astra result; the source does not provide a full item-by-item dataset for every configuration. Meyer says the robustness run is a more reliable guide, but the results have not been independently verified in the material provided.
It is also unclear how the models would perform across other kinds of news, longer investigations, breaking coverage, or different prompts and editorial checks. The report gives average scores and intervals, but does not establish that the same cost and quality trade-offs would apply to other publishers. Meyer’s description of judge scoring is not a substitute for a detailed account of human editorial review, which is not provided here.
The report says two fallback models were removed on Oct. 7, but its final account of the new failover chain is incomplete: it states that GPT-6 Luna now runs first and repeats Luna in the next phrase. The identity of any replacement backup model is not clear from the supplied material.
As an affiliate, we earn on qualifying purchases.
The Next Pipeline Checks
Meyer’s report says GPT-6 Luna is now the primary model in the news route and that the two problematic fallbacks were removed on Oct. 7. The next practical test is whether Luna continues to meet the pipeline’s quality, latency and cost requirements across a wider range of incoming stories. The report does not give a scheduled date for another benchmark or say whether further fallback models have been selected.
For readers assessing the findings, the next useful evidence would include more source types, a larger article sample and clearer details on editorial checks, as well as a complete account of the active failover chain. Until those details are available, the scores are best read as results from Meyer’s own production-oriented test rather than a universal ordering of AI writing models.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which model scored highest in Meyer’s test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It scored 3.93 across 12 articles in the reported robustness run.
Why did Meyer choose GPT-6 Luna for production?
Meyer says Luna scored 3.67 in the robustness run, produced valid articles in every reported attempt, and cost about one-hundredth as much per article as Astra. The choice reflects his pipeline and quality threshold.
Did higher reasoning settings improve Luna’s articles?
Not reliably in the reported test. Meyer says the score differences were within the noise, while the highest setting took about 211 seconds and cost about 12 times as much as the no-effort setting.
What did the test find about fallback models?
Meyer reports that GPT-4.1 nano produced invented quotations and scored 1.00 for source fidelity, while locally run gpt-oss-120b produced no valid articles in four attempts. He says both were removed from the route on Oct. 7.
Do these results show which model is best for every newsroom?
No. The findings are based on Meyer’s prompt, sources, languages and pipeline. The report does not establish that the same ranking or costs apply to other publishers or news tasks.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
