A reproducible 300-question evaluation across medical exams, biomedical literature, multilingual question answering, and clinical calculation
Meta released Muse Glimmer on 10 August 2026: thirty billion parameters, open weights, built for agentic work on consumer hardware.[1,2] I tested a quantized version five days later on a single NVIDIA RTX 3090 — partly because it was the first release in months whose size and quantization looked like a comfortable fit for 24 GB of VRAM rather than a fight with it, partly because I had a few free hours.
Whether a language model can replace clinical judgment is not something a benchmark settles, and I did not set out to ask it. My question was narrower: how accurately and reliably can this new general-purpose model answer a diverse set of established medical benchmark questions when it runs entirely on a personal workstation? That much a benchmark can address, at least in part.
Encouraging, and uneven. Muse Glimmer answered 248 of 300 questions correctly, for a pooled accuracy of 82.7% (Wilson 95% confidence interval, 78.0%–86.5%). Its macro-average across six benchmark families was 82.8%. Every response came back in the required structured format :no invalid outputs, no truncated generations, no parsing failures.
The headline number hides a clinically important pattern, though, and I should also say at the outset that I distrust benchmark scores this high. Both points I come back to below. Performance reached 98% on the MedQA sample and 92.5% on MedExpQA, including 90% on the Italian subset. It fell to 74% on clinical calculation and PubMedQA, and to 70% on MedMCQA. Inside MedCalc-Bench, formula-based physical and laboratory calculations were usually correct; rule-based risk scores were much less reliable. That contrast matters more than the overall ranking.
Key result: Muse Glimmer 30B Q4KM achieved 82.7% accuracy on 300 medical benchmark questions when run locally on an RTX 3090, with 0 invalid responses and a median end-to-end latency of 6.94 seconds per question.
Why test a local general-purpose model on medicine?
Medical language-model evaluations have traditionally focused on very large proprietary systems, or on models specifically adapted to medicine. MultiMedQA helped establish the modern pattern of combining examination questions, biomedical research questions, and consumer health tasks in a single broader framework.[3] Newer benchmarks have pushed into multilingual assessment, clinical calculations, electronic health records, and real-world workflows.
Local models raise a different practical question, and it is the one I have been working on for some time. A model that runs on one workstation can be evaluated without sending a single prompt to a cloud service. Infrastructure costs are predictable. It works without network access, and it can be dropped into experimental workflows where local control matters. None of that establishes clinical safety — it does make independent, reproducible testing considerably easier.
What makes Muse Glimmer interesting is precisely that Meta did not release it as a medical model. It is a dense 30-billion-parameter causal language model, distilled from a larger teacher, optimized for agentic tasks, tool use, multimodal understanding, and long-running local workflows.[1,2] Testing it on medicine therefore asks whether general reasoning capability transfers into a demanding domain that was never the stated target.
Methods
Model and local hardware
The evaluated model was Meta Muse Glimmer 30B in GGUF Q4KM quantization. Inference ran locally on Windows 11, on an NVIDIA RTX 3090 with 24 GB of VRAM and 64 GB of system RAM. The model was served through the OpenAI-compatible endpoint of llama-server, using the NVIDIA CUDA build of llama.cpp version 0.1.0-dev, build 10437, commit 16d222fc5.[4]
Runtime context length was 8,192 tokens. All model layers were offloaded to the GPU. Observed model allocation was approximately 18.0 GB of VRAM, which left enough headroom for the context and runtime buffers.
I heard the thermal problem before I measured it. During an initial pilot at the card’s 350 W default limit the fans went to 100% and stayed there; when I finally looked, the core had settled at 88–89 °C. I stopped the run, discarded those responses, and restarted from zero with a fixed 280 W power limit, 80% of the default board limit. Under sustained inference the core then stabilized around 80–82 °C, with fan speed near 68% in the monitored block. The final 300-question dataset therefore contains only responses generated under the 280 W configuration.
Benchmark composition
The test set was a reproducible stratified sample of 300 questions generated with sampling seed 20260813, combining six established benchmark families:
| Benchmark | Questions | Domain |
|---|---|---|
| MedQA | 50 | US medical licensing-style multiple-choice questions |
| MedMCQA | 50 | Multi-subject Indian medical entrance-examination questions |
| PubMedQA | 50 | Yes/no/maybe inference from biomedical research abstracts |
| MedCalc-Bench Verified | 50 | Rule-based and equation-based clinical calculations |
| MedExpQA | 40 | Medical exam questions with gold explanations: 20 English and 20 Italian |
| MMLU medical subsets | 60 | Ten questions from each of six medical and biological subjects |
| Total | 300 |
MedQA was introduced as a multilingual open-domain medical examination dataset.[5] MedMCQA contains more than 194,000 questions drawn from AIIMS and NEET-PG examinations across 21 medical subjects.[6] PubMedQA tests whether a model can infer a yes, no, or maybe answer from biomedical research text.[7] The MMLU subsets covered anatomy, clinical knowledge, college biology, college medicine, medical genetics, and professional medicine.[8]
Two features drew me to MedExpQA, both of them scarce in medical benchmarks: multilingual questions, and gold explanations written by physicians.[9] MedCalc-Bench asks for something different again — extracting clinical variables, then applying equations or rule-based scores.[10] I used the maintained MedCalc-Bench Verified dataset rather than the deprecated earlier versions, for reasons that turn out to matter (see below).[11]
Inference protocol
Inference was zero-shot and sequential. During generation the runner had access only to the question file; the answer key and scoring records stayed separate until inference had finished. Principal generation settings:
| Parameter | Value |
|---|---|
| Temperature | 0 |
| Top-p | 1 |
| Seed | 42 |
| Maximum completion tokens | 2,048 |
| Reasoning strength | Low |
| Context length | 8,192 |
| Timeout | 180 seconds |
| Retries | 2 |
| Concurrent requests | 1 |
A machine-readable final answer was required by the output contract. Multiple-choice datasets returned the selected option, PubMedQA returned yes, no, or maybe, and MedCalc-Bench returned a numerical or date result. Responses were written to JSONL immediately, so that an interrupted run could resume without repeating successful questions.
For thermal control and operational recovery the evaluation ran in six blocks of 50 questions, with the card allowed to fall below 50 °C between blocks before the next one started. Overcautious, probably. It cost a few minutes per block and removed a variable I did not want to be arguing about afterwards. Model, prompt, generation parameters, power limit, and output file were identical across blocks.
Scoring and statistical analysis
MedQA, MedMCQA, PubMedQA, MedExpQA, and MMLU were scored by normalized exact match. MedCalc-Bench decimal and integer outputs counted as correct when the parsed numerical value fell within the inclusive lower and upper limits supplied by the verified source dataset; date outputs used the corresponding inclusive date interval.
Primary descriptive outcomes were pooled accuracy and macro-average accuracy across benchmark families, with Wilson 95% confidence intervals for proportions. The macro-average weights each of the six families equally; pooled accuracy weights every question equally. Operational outcomes were invalid-response rate, end-to-end latency, and completion-token throughput.
Results
Overall performance
Muse Glimmer answered 248 of 300 questions correctly. Pooled accuracy was 82.7%, the equal-weighted macro-average 82.8%. That the two values sit almost on top of each other indicates the overall result was not materially driven by the slightly different sample sizes between datasets.
All 300 generations carried status: ok, finish_reason: stop, and a valid parsed answer. No duplicated question identifiers. Invalid-response rate: 0%.

Accuracy differed substantially by benchmark
| Dataset | Correct / N | Accuracy | Wilson 95% CI |
|---|---|---|---|
| MedQA | 49/50 | 98.0% | 89.5%–99.6% |
| MedExpQA | 37/40 | 92.5% | 80.1%–97.4% |
| MMLU medical | 53/60 | 88.3% | 77.8%–94.2% |
| MedCalc-Bench | 37/50 | 74.0% | 60.4%–84.1% |
| PubMedQA | 37/50 | 74.0% | 60.4%–84.1% |
| MedMCQA | 35/50 | 70.0% | 56.2%–80.9% |
MedQA produced the strongest result by a distance: one miss in fifty. MedExpQA and the medical MMLU subsets also gave high point estimates. MedMCQA was weakest, with 15 errors, while MedCalc-Bench and PubMedQA contributed 13 each.
Between them, MedMCQA, MedCalc-Bench, and PubMedQA accounted for 41 of the 52 total errors (78.8%). Such a concentration suggests that an overall medical QA score is not enough to characterize model behaviour. Examination-style recall, literature inference, and clinical computation stress genuinely different capabilities.
English and Italian MedExpQA
Nineteen of 20 English MedExpQA questions were answered correctly (95%), and 18 of 20 Italian ones (90%). Five points of difference across 20 questions per language is not a finding; the confidence intervals are wide and strongly overlapping, and I would not build anything on it.
The Italian result is still useful in a weaker sense. It shows the model’s strong MedExpQA performance was not confined to English in this sample. A proper multilingual evaluation would need the complete MedExpQA test set and prespecified paired language comparisons.
| MedExpQA language | Correct / N | Accuracy |
|---|---|---|
| English | 19/20 | 95.0% |
| Italian | 18/20 | 90.0% |
MMLU medical subjects
Across the six MMLU subsets, performance ranged from 70% to 100%:
| MMLU subject | Correct / N | Accuracy |
|---|---|---|
| College biology | 10/10 | 100% |
| Medical genetics | 10/10 | 100% |
| Clinical knowledge | 9/10 | 90% |
| Professional medicine | 9/10 | 90% |
| College medicine | 8/10 | 80% |
| Anatomy | 7/10 | 70% |
Descriptive only, these. With ten questions per subject a single answer moves the estimate ten percentage points, so the ordering should not be read as a ranking. The distribution is nevertheless helpful for choosing which domains deserve larger confirmatory samples: anatomy and college medicine are the natural targets for error analysis.
Clinical calculation exposed a specific weakness
Aggregate MedCalc-Bench accuracy was 74%. The category-level breakdown is sharply heterogeneous:
| MedCalc category and output | Correct / N | Accuracy |
|---|---|---|
| Physical, decimal | 13/13 | 100% |
| Laboratory test, decimal | 16/17 | 94.1% |
| Dosage, integer | 2/3 | 66.7% |
| Severity, integer | 2/3 | 66.7% |
| Risk, integer | 4/12 | 33.3% |
| Risk, decimal | 0/2 | 0% |
Subgroup sizes here are small, particularly for dosage, severity, and decimal risk. The error pattern is clinically plausible all the same, and worth pursuing. Continuous formula-based calculations went well; rule-based risk scores, which require correctly extracting several criteria, assigning points, and summing them, did not. One case was a CHA₂DS₂-VASc calculation where the model returned 3 against a verified score of 5.
Worth pausing on that one. Both values sit above the usual anticoagulation threshold, so in that particular case the downstream decision would probably not have changed — which is exactly the kind of near-miss that makes an aggregate accuracy figure feel reassuring and tell you nothing. I have not yet gone back through the raw outputs to identify which criteria the model dropped, and until someone does, “74% on clinical calculation” is a number without a mechanism behind it.
The wider point is that numerical ability should not be treated as one construct. Applying a direct equation to explicit measurements is a different task from identifying several clinical attributes buried in prose and mapping each to a scoring rule. A larger analysis would want to classify MedCalc errors into entity extraction, rule selection, arithmetic, unit conversion, and final-answer formatting.
Latency and throughput
Median end-to-end latency was 6.94 seconds per question, with a 90th percentile of 16.77 seconds. Median completion throughput, calculated end to end, was 29.96 tokens per second.
MedCalc-Bench demanded the longest responses and the most inference time — median latency 16.51 seconds, against roughly five to eight seconds elsewhere.
| Dataset | Median latency | P90 latency |
|---|---|---|
| MedMCQA | 5.14 s | 10.98 s |
| MMLU medical | 5.16 s | 9.55 s |
| PubMedQA | 5.55 s | 8.05 s |
| MedQA | 7.38 s | 13.98 s |
| MedExpQA | 8.13 s | 13.51 s |
| MedCalc-Bench | 16.51 s | 26.06 s |

Capping the board power reduced thermal stress while preserving practical throughput — a trade I would make again. None of these latency figures should be assumed to generalize to other quantizations, context sizes, GPU models, drivers, or inference-engine versions.
What the 82.7% score does (and does not) mean
Read correctly, the result says this: Muse Glimmer achieved 82.7% accuracy on a specific stratified sample of public medical benchmarks, under a documented local inference configuration.
It does not say the model has 82.7% diagnostic accuracy, that it is safe for clinical use, or that it can replace a clinician. Multiple-choice examinations and constrained final-answer tasks capture a thin slice of the clinical process. They do not test longitudinal synthesis, uncertainty communication, patient preferences, physical examination, workflow integration, or what happens when a recommendation is wrong.
Which brings me to the 98% on MedQA, and to a wider unease. I am always suspicious when a result is this good. I have argued elsewhere that a good deal of medical AI evaluation is quietly circular: the corpora used to train these systems and the corpora used to test them overlap, nobody outside the developing lab can say by how much, and the leaderboard rewards precisely that overlap. The other half of the pattern is just as visible — systems that look excellent on public benchmarks routinely perform considerably worse once they meet real clinical data. [Link to your earlier article on medical AI evaluation goes here.]
MedQA fits the description almost too neatly. Public, heavily reused, old enough to have been absorbed several times over. Meta does not disclose the complete training dataset, so contamination cannot be excluded here either. Genuine transfer, familiarity with the format, outright memorization: the 98% is compatible with all three, and from outside there is no way to separate them. Recent benchmark audits make a similar argument in more measured language, adding label quality and weak governance to the list of things that inflate medical leaderboard results.[12]
The same caution applies to comparing this run against published model scores. Prompt format, answer extraction, dataset version, quantization, inference framework, tool access — each of them moves performance. MedCalc-Bench is especially sensitive to version and scoring choices, since deprecated releases contained corrected labels and implementations. Hence the Verified dataset.[10,11]
Strengths
This pilot has several methodological strengths:
- The answer key was isolated during inference. The runner read only the question file, and predictions were joined to gold answers afterwards.
- The sample was reproducible. Dataset allocation and sampling seed were fixed in advance.
- Inference was local and sequential. No cloud model, no web search, no retrieval system, no external clinical calculator.
- Every raw output was preserved. Question-level auditing remains possible, rather than reliance on an aggregate score.
- Scoring was deterministic. Normalized exact match for multiple-choice tasks; source-provided limits for MedCalc.
- Operational reliability was measured, not assumed. All 300 responses terminated normally and parsed successfully.
- The deployment configuration was recorded in full. Model quantization, GPU, runtime, context, power limit, decoding settings.
Limitations
The evaluation also has important limitations, and some of them are severe.
Three hundred questions are enough for a technical pilot and nowhere near enough for definitive model ranking. Dataset-specific confidence intervals stay wide, and the language and subject subgroups are far smaller still. My complete local benchmark contains 8,895 records and would support substantially more precise estimates; I simply did not have the GPU hours.
Then there is the fact that one model, one quantization, one hardware configuration, and one inference engine were tested. The result belongs to the whole deployment configuration, not to some abstract base model. Q4KM quantization may itself alter accuracy relative to higher-precision weights, and by how much I cannot say from a single run.
The prompt included Muse Glimmer’s low reasoning-strength instruction. Appropriate for a single-model pilot; not a neutral prompt for cross-model comparison. Any comparative study needs a model-agnostic primary prompt frozen before a second model is evaluated, with model-specific recommended prompts reserved for a secondary “best configuration” analysis.
Deterministic decoding was used once per question, and only once. Repeatability should be confirmed on a prespecified subset even at temperature zero — I would not assume it.
More fundamentally, this study scored final answers rather than the clinical validity of reasoning. A correct answer can sit on top of flawed reasoning, and an incorrect benchmark answer occasionally reflects an ambiguous or plainly wrong gold label. Separating knowledge deficits, extraction failures, arithmetic mistakes, questionable labels, and clinically unsafe reasoning would require blinded physician review.
And public benchmark performance says nothing at all about calibration, abstention, hallucination in open-ended responses, demographic bias, guideline currency, or real-world patient safety.
From rapid benchmark to research programme
This started as a rapid evaluation of a newly released model, run in the gaps of a working week. The result supports something more rigorous: a paired study of open-weight models that fit on a single 24 GB consumer GPU.
A confirmatory protocol should be frozen before the next model is touched. It needs to specify model inclusion criteria, exact dataset versions, a neutral prompt, quantization, context length, power limit, inference-engine build, primary and secondary outcomes, multiplicity correction, and the handling of invalid responses. All models answer the same questions, which permits paired comparison with McNemar tests and stratified bootstrap confidence intervals for accuracy differences.
Blinded clinical error review would be the strongest extension. Two clinicians, independently classifying a stratified sample of discordant or incorrect responses into knowledge, interpretation, entity extraction, rule selection, arithmetic, unit, and benchmark-label errors. That moves the work off the leaderboard and towards an account of where locally deployable models fail.
Conclusion
Muse Glimmer 30B delivered strong medical benchmark performance on a single consumer GPU: 248 of 300 questions correct, no invalid outputs, median latency below seven seconds at a 280 W power limit. Its best results came from MedQA, MedExpQA, and the medical MMLU subsets. Its worst came from MedMCQA, and from tasks requiring biomedical inference or rule-based clinical risk calculation.
The 82.7% is the least interesting number in this article. What the run actually showed is that a single aggregate score conceals a jagged capability profile — highly capable on familiar medical examination formats, considerably less reliable once clinical information had to be converted into a multivariable risk score. Which is the argument for multidomain, versioned, question-level evaluation, and for treating any headline medical benchmark figure, this one included, with more suspicion than it usually receives.
Frequently asked questions
Can Muse Glimmer 30B run on an RTX 3090?
Yes. The Q4KM GGUF used here occupied approximately 18 GB of the RTX 3090’s 24 GB VRAM at an 8,192-token context. Exact requirements depend on quantization, context, runtime buffers, and any optional multimodal components.
What medical benchmark score did Muse Glimmer achieve?
82.7% pooled accuracy and 82.8% macro-average accuracy on this stratified 300-question sample. The 95% Wilson confidence interval for pooled accuracy was 78.0%–86.5%.
Was the model connected to the internet or external tools?
No. Inference was local, sequential, and closed-book — no web search, no retrieval-augmented generation, no clinical calculator.
Does this result show that Muse Glimmer is clinically safe?
No. It measures performance on public benchmark questions. Not diagnostic safety, not treatment quality, not calibration, and not performance in patient care.
Why was the GPU power limited to 280 W?
The stock 350 W pilot produced sustained core temperatures of 88–89 °C at maximum fan speed, so I discarded it. Restarting at 280 W brought the sustained core temperature down to roughly 80–82 °C while retaining practical inference speed. The definitive run was also split into six blocks of 50 questions, with the card cooling below 50 °C between blocks.
References
- Meta AI. Introducing Muse Glimmer: An Open Agentic Model That Runs on a Single GPU. Published 10 August 2026.
- Meta. Muse Glimmer 30B model card. Hugging Face; 2026.
- Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620:172–180. doi:10.1038/s41586-023-06291-2.
- Gerganov G, llama.cpp contributors. llama.cpp: LLM inference in C/C++. GitHub.
- Jin D, Pan E, Oufattole N, Weng WH, Fang H, Szolovits P. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences. 2021;11(14):6421. doi:10.3390/app11146421.
- Pal A, Umapathi LK, Sankarasubbu M. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical Domain Question Answering. Proceedings of the Conference on Health, Inference, and Learning. 2022;174:248–260.
- Jin Q, Dhingra B, Liu Z, Cohen WW, Lu X. PubMedQA: A Dataset for Biomedical Research Question Answering. Proceedings of EMNLP-IJCNLP. 2019:2567–2577. doi:10.18653/v1/D19-1259.
- Hendrycks D, Burns C, Basart S, et al. Measuring Massive Multitask Language Understanding. International Conference on Learning Representations. 2021.
- Alonso I, Oronoz M, Agerri R. MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering. Artificial Intelligence in Medicine. 2024;155:102938. doi:10.1016/j.artmed.2024.102938.
- Khandekar N, Jin Q, Xiong G, et al. MedCalc-Bench: Evaluating Large Language Models for Medical Calculations. Advances in Neural Information Processing Systems. 2024;37.
- Khandekar N, et al. MedCalc-Bench Verified. Maintained dataset repository; accessed August 2026.
- Chen W, Yu G, Cheung YF, et al. Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models. Accepted at ACL 2026; arXiv:2508.04325.









