More Expensive, Slower but Actually Better?
Every frontier release arrives with the same implicit promise. Better reasoning, better reliability, better behaviour on the hard cases. Higher cost too, of course, which tends to be presented as the price of that improvement rather than as something anyone is expected to verify.
When GPT-6 Astra became available I wanted to test the promise against a deliberately narrow question:
Is Astra actually better than GPT-5.6 Sol at medical question answering?
The natural move would have been to build a fresh benchmark around the new model. I did the opposite, and reused a set of 300 medical questions frozen some time earlier for a different purpose entirely: evaluating two locally deployed models, Muse Glimmer 30B and Qwen3.8 27B.
That accident of reuse turned out to be the most useful design decision in the whole experiment. It was no longer Astra versus Sol. It became a small four-way comparison between two frontier cloud systems and two models that fit on a single RTX 3090.
The result is not what the version numbers — or the invoices — would lead you to expect.
The short answer
Across all 300 questions:
| Model | Correct | Accuracy |
|---|---|---|
| GPT-6 Astra | 261/300 | 87.0% |
| GPT-5.6 Sol | 258/300 | 86.0% |
| Muse Glimmer 30B | 248/300 | 82.7% |
| Qwen3.8 27B | 215/300 | 71.7% |
Astra finished first. Barely.
Three more correct answers than Sol, out of three hundred — an absolute advantage of one percentage point. Because every model answered exactly the same items in exactly the same order, the two frontier systems can be compared question by question rather than only in aggregate: both correct on 248 questions, Astra alone correct on 13, Sol alone correct on 10, both wrong on 29.
The exact McNemar test on those discordant pairs gave p = 0.678, and the bootstrap 95% confidence interval for the accuracy difference comfortably crossed zero.
So this benchmark provides no convincing evidence that Astra is medically more accurate than Sol. It may well be. The data simply cannot separate an advantage of this size from sampling variability, and it would take a considerably larger question set to try.
A deliberately controlled benchmark
The benchmark consisted of 300 fixed questions drawn from six sources: MedQA, MedMCQA, PubMedQA, MedCalc-Bench, MedExpQA, and the medical subsets of MMLU. Same question IDs, same order, every model. Of these, 280 were in English and 20 in Italian.
This was never designed as a clinical validation study, and it should not be read as one. It is better understood as a controlled technical benchmark of medical question-answering capability — which is a much narrower thing than it sounds.
The distinction matters more than the numbers. Performance on public benchmarks says nothing directly about clinical competence, safe deployment, or generalisation to a real patient. Contamination is also a live concern: some of these items, or material closely related to them, may well sit somewhere in the training corpora of the models being tested. There is no way for me to exclude it.
What the design does allow is a fair comparison under comparable conditions. That was the whole point.
Scoring turned out to be the hard part
One methodological problem surfaced almost immediately: comparing LLMs is not a matter of checking whether two output strings match.
A model may answer A. Another returns 1. A third writes out the full text of the first option. If all three refer to the same choice, marking two of them wrong measures formatting compliance, not medical knowledge — and formatting compliance is precisely the kind of variable that quietly ruins benchmark comparisons.
For the choice-based datasets I therefore normalised answers by option position, so that semantically equivalent representations mapped to the same response.
MedCalc-Bench needed something else. The original dataset supplies an acceptable Lower Limit–Upper Limit interval for each numerical answer, and I reproduced the scoring rule from my earlier local-model benchmark: extract the first numerical result in the model’s response, count it correct if it falls inside that inclusive interval.
Trivial in principle. Less trivial with an answer like:
48 mL/min/1.73 m²
where 48 is the calculation result and 1.73 belongs to the unit. Naive numeric extraction gets this wrong in a way that is easy to miss and hard to notice once it is buried in an aggregate score.
All four models were rescored from the canonical benchmark records. I deliberately did not reuse the stored correctness labels from the earlier local-model run.
Overall accuracy: Astra wins, and it hardly matters
The ranking came out as Astra 87.0% → Sol 86.0% → Glimmer 82.7% → Qwen3.8 71.7%, which at first glance looks like exactly the hierarchy anyone would have predicted.
The magnitudes are more informative than the ordering. Astra exceeded Sol by 1.0 percentage point, with a paired bootstrap interval of roughly −2.0 to +4.3 points, and the two models disagreed in outcome on only 23 of 300 questions.
That is not evidence of superiority. It is a tie within the resolution of the experiment.
The individual datasets tell a messier story
Pooling everything into a single accuracy figure hides a good deal of variation.
| Dataset | Astra | Sol | Glimmer | Qwen3.8 |
|---|---|---|---|---|
| MedCalc-Bench | 88.0% | 86.0% | 74.0% | 22.0% |
| MedExpQA | 97.5% | 92.5% | 92.5% | 87.5% |
| MedMCQA | 84.0% | 86.0% | 70.0% | 62.0% |
| MedQA | 98.0% | 96.0% | 98.0% | 92.0% |
| MMLU medical | 95.0% | 96.7% | 88.3% | 86.7% |
| PubMedQA | 60.0% | 58.0% | 74.0% | 80.0% |
No model dominates every category. Astra was strongest on MedCalc-Bench and MedExpQA; Sol edged ahead on MedMCQA and MMLU; Glimmer matched Astra’s 98% on MedQA while running on a desktop GPU.
And then there is PubMedQA.
PubMedQA inverts the whole ranking
This was, for me, the single most interesting result in the experiment:
- Qwen3.8: 80%
- Glimmer: 74%
- Astra: 60%
- Sol: 58%
The weakest model overall became the strongest on this dataset, and the two frontier systems ended up twenty points behind it.
I do not have a clean explanation. Dataset construction, answer format, the yes/no/maybe structure, training exposure, alignment behaviour on abstract-level reasoning — any of these could interact with model architecture in ways that a pooled leaderboard number simply erases. What the result does establish is that a model looking clearly weaker in aggregate may nevertheless be substantially better on a specific task family, and that a frontier model does not inherit dominance across every medical benchmark just because its overall score is higher.
Reducing benchmark results to one number is convenient. It is also how you end up choosing the wrong model for your actual workload.
What those three extra correct answers cost
The accuracy difference becomes considerably more interesting once compute cost enters the picture. For these 300 questions:
| Metric | Sol | Astra |
|---|---|---|
| Total API cost | $0.96 | $2.96 |
| Cost per correct answer | $0.0037 | $0.0113 |
| Median latency | 1.85 s | 2.47 s |
| 90th percentile latency | 5.57 s | 8.39 s |
| Mean reasoning tokens | 101 | 138 |
Astra cost a little over three times as much. Median latency was about 33% higher, p90 latency roughly 51% higher, and it consumed appreciably more reasoning tokens along the way.
The return on all of that was three additional correct answers out of 300.
None of which makes Astra a bad model, and it does not establish Sol as the better choice in general — there are almost certainly harder tasks, outside anything this benchmark touches, where the extra reasoning capacity earns its keep. But for this particular medical workload the trade-off is difficult to argue away. If the objective is answering questions of this kind, Sol offers a markedly better accuracy-per-dollar ratio, and it is not close.
Reasoning can eat the entire answer budget
One operational finding deserves its own note, because I did not anticipate it.
Both frontier models ran with a 1,024-token output budget and high reasoning effort. In a handful of cases the model spent that entire allowance internally and produced no visible final answer: 5 times with Astra, 4 times with Sol. These were classified separately as output-budget failures and counted as incorrect in the primary analysis.
The phenomenon illustrates a failure mode that is easy to overlook in reasoning models. More internal deliberation is not automatically better if the deliberation leaves no room for the answer.
It has a practical consequence as well. Output-token limits used to be a presentation setting. With reasoning models they become part of the effective inference configuration — and therefore part of the benchmark methodology, whether or not anyone reports them.
And then there is Glimmer
The number I keep returning to is not Astra’s 87%. It is Glimmer’s 82.7%.
Muse Glimmer 30B ran entirely locally, on a single RTX 3090: 4.3 points below Astra, 3.3 points below Sol. In the paired comparison against Sol, the exact McNemar test did not reach conventional significance in this 300-question sample, and Astra versus Glimmer produced a similarly modest difference despite an enormous gap in deployment cost and infrastructure.
This should not be overinterpreted, and I want to be explicit about why. Cloud frontier models and a local 30B model differ along many dimensions that 300 multiple-choice questions do not probe at all — context handling, tool use, robustness on open-ended clinical text, behaviour under adversarial phrasing. Latency is not comparable either, since local inference and remote API inference sit on entirely different infrastructure.
Still. A model that can be stored and executed on consumer-grade hardware reached 82.7% against 86–87% for current frontier cloud systems on identical questions. That gap is narrower than most people would guess, and narrower than I would have guessed before running it.
Qwen3.8 shows the other face of local inference
Qwen3.8 27B managed 71.7% overall, and its behaviour was far less consistent across datasets: 92% on MedQA, 87.5% on MedExpQA, 86.7% on medical MMLU — and 22% on MedCalc-Bench.
That last figure is the one that matters operationally. Whatever is happening in the numerical-reasoning pathway, it is not a small degradation; it is a collapse.
The model was extremely fast in this configuration, with a median latency around 0.82 seconds, but speed did not compensate for anything. It also produced 19 unparseable responses, against perfect structured-output compliance from Glimmer.
Local deployment, then, is not a single trade-off against the cloud. The choice of local model matters enormously, and the spread between two 27–30B models here is wider than the spread between the best local model and the frontier.
So what does Astra buy you?
On this benchmark, remarkably little. Against Sol: three additional correct answers and one percentage point of accuracy, for roughly 3.1× the API cost, 33% higher median latency, and substantially more reasoning tokens.
There may well be domains where Astra’s additional reasoning capacity becomes decisive. This benchmark did not find them. For routine medical benchmark questions the two models behaved like closely matched systems rather than members of different capability tiers.
The most useful result is the absence of a dramatic one
New releases invite winner-versus-loser narratives, and controlled experiments have an inconvenient habit of refusing to supply them.
Astra did finish first. But the margin is small enough that calling it better at medicine would go beyond what these data support. The defensible version reads less well and is more accurate:
On this fixed 300-question medical benchmark, GPT-6 Astra and GPT-5.6 Sol achieved essentially comparable accuracy, while Astra required substantially greater API expenditure and longer response times.
That is not a disappointing outcome. It is the reason for benchmarking against fixed datasets in the first place. Without a controlled comparison, improvement is something you assume. With one, it has to be demonstrated — and sometimes it declines to be.
A four-model perspective
The experiment reads better as a spectrum than as a race:
Astra — 87.0% Sol — 86.0% Glimmer — 82.7% Qwen3.8 — 71.7%
The top three are separated by 4.3 percentage points while representing radically different deployment strategies: a new premium cloud reasoning model, a cheaper frontier cloud model, and a 30B model on one consumer GPU.
Qwen3.8 demonstrates that local deployment guarantees nothing. Glimmer demonstrates that it can nonetheless come surprisingly close. And Astra demonstrates that a newer, considerably more expensive model does not necessarily deliver a proportionate improvement on any given specialised task.
Limitations
Several caveats should keep all of this in proportion.
Three hundred questions is a small benchmark — adequate for detecting large differences, inadequate for establishing equivalence between closely matched models. The questions come from publicly available datasets, so training-data exposure cannot be excluded and probably should be assumed to some degree. What is measured here is question-answering accuracy, not clinical judgment, patient safety, longitudinal reasoning, tool use, or real-world decision making.
The dataset-level analyses rest on only 40–60 questions each and are descriptive at best; I would not build an argument on any single cell of that table. Cloud API latency depends on remote infrastructure, load and network conditions, and is therefore not directly comparable with locally measured latency.
Finally, everything reported here reflects one inference configuration — high reasoning effort, a 1,024-token output budget for Astra and Sol. Different settings would plausibly change both accuracy and efficiency, and I have not explored that space.
Conclusion
GPT-6 Astra produced the highest score on this 300-question medical benchmark. The margin over GPT-5.6 Sol was 261 versus 258 correct answers — neither statistically convincing nor large enough to establish meaningful medical superiority. Astra also cost roughly three times as much and responded more slowly.
Meanwhile a locally deployed 30B model reached 82.7%, a few points behind both frontier cloud systems.
Which suggests the more useful question is not is Astra better than Sol? but rather:
How much additional capability are we actually receiving for each additional unit of compute, latency and cost?
For this particular medical workload, the answer appears to be: not very much.