Open-Weight Medical LLMs

There is a specific moment, familiar to anyone who has gone down this road, when you finish downloading a model with “Med” in its name and realise you have no idea whether it is actually any good. The name promises domain expertise. The model card promises benchmarks. Neither tells you what happens when you ask it something a cardiac surgery registrar would ask.

This is the map I wish I had before I started. It covers the open-weight models that have been explicitly tuned for medicine, what each one is built on, what it weighs on disk, what licence it carries, and what its published numbers actually say. It also covers what those numbers do not say, which turns out to be the more interesting half of the story.

One thing to settle before we go further. None of these models is a clinical tool. Every serious model card says so, in language that is worth taking literally rather than treating as boilerplate. What they are is raw material: starting points for research, for experimentation, and for the kind of private workflows where the data never leaves your machine.

MedGemma: the one with the numbers

If there is a centre of gravity in this landscape, it is Google’s MedGemma, built on the Gemma 3 architecture and distributed under the Health AI Developer Foundations programme. The collection comes in a 4B multimodal version, a 27B text-only version, and a 27B multimodal version, and Google is explicit that the multimodal variants use a SigLIP image encoder trained on de-identified medical data.

The January 2026 refresh, MedGemma 1.5, updated the 4B model rather than the 27B, and the changes matter for anyone thinking about clinical text rather than images. Beyond the imaging additions (3D CT and MRI volumes, whole-slide histopathology, longitudinal chest X-ray comparison), the update targeted exactly the tasks a hospital actually drowns in: extraction of structured data from unstructured lab reports, and interpretation of text-based EHR data. The reported gains are real but modest on the reasoning side and dramatic on extraction: MedQA went from 64% to 69%, while lab report extraction F1 jumped from 60% to 78% and EHR question answering reached 90%.

On raw medical knowledge the 27B text model is the strongest thing in this article: it scores 87.7% on MedQA, which the team notes is within three points of DeepSeek R1 at roughly one tenth the inference cost. Google’s own guidance is that for most use cases the 27B will yield the best performance, and that is worth remembering when you are tempted by the convenience of the 4B.

The caveats are stated by Google with unusual candour. The models are not clinical grade and will likely require further fine-tuning. There is an explicit warning about data contamination, the possibility that the model has already seen related medical content in pre-training, which means published benchmarks may overstate its ability to generalise to genuinely novel cases. Developers are told to validate on datasets that are not publicly available. How many people downloading a GGUF this weekend will do that?

That warning deserves more weight than it usually gets. An 87.7% on MedQA is a number about MedQA. It is not a number about the patient in bed 4.

Licensing is the practical friction point: the weights are open but governed by the Health AI Developer Foundations terms of use, which must be accepted before download. The community requantised GGUF repositories generally spare you that step, but you are still bound by the terms.

Meditron: the guideline-trained model with an uncomfortable record

Meditron comes out of EPFL and Yale and has an unusually principled origin story. The original Meditron-7B and Meditron-70B were adapted from Llama-2 through continued pre-training on a curated medical corpus that included PubMed papers and abstracts and, distinctively, a purpose-built dataset of internationally recognised clinical practice guidelines. The later generation moved to Llama-3.1 as a base, giving the Meditron3-8B that most people will encounter today.

The published numbers were competitive when they landed. Meditron-70B beat Llama-2-70B and GPT-3.5 on several medical reasoning tasks, and the team was careful to note it was not adapted to deliver that knowledge safely or within professional constraints, recommending against clinical use without randomised testing in real settings.

Then the independent evaluations arrived, and they are sobering. On a benchmark built from Israeli neurology board certification exams, Meditron-70B achieved 52.9% base accuracy, the lowest among all 70B models evaluated, against 69.5% for LLaMA 3.3-70B and 65.9% for OpenBioLLM-70B. Worse, its performance degraded further under retrieval-augmented generation, dropping to 41.2%, which the authors read as an incompatibility between the model’s internal representations and external evidence. A medical model that gets worse when you hand it the guidelines is not a comfortable finding for a project whose distinguishing feature was training on guidelines.

I want to be careful here, because this is one benchmark, in one language, on one specialty that is not the one Meditron was optimised for, and a single unflattering result is not a verdict. Or more precisely: it is not a verdict on the model, but it is a verdict on our confidence. Something in that pipeline is not doing what the model card implies it does.

There is a newer and more interesting Meditron story, though. In 2026 the team published Fully Open Meditron, an auditable pipeline where the corpus, the code, and the training recipe are all disclosed, and fine-tuned it onto fully open bases including Apertus and EuroLLM. The resulting Apertus-70B-MeditronFO is the strongest fully open medical model at 53.77 average across benchmarks, narrowing but not closing the gap to MedGemma-27B at 60.67. Every MeditronFO variant improved over its base, with gains ranging from +0.66 for EuroLLM-22B to +12.80 for Apertus-8B, and smaller bases benefiting most.

That last detail is the most useful thing in this entire article, and I will come back to it.

OpenBioLLM: the benchmark champion with a nasty asterisk

OpenBioLLM from Saama AI Labs comes in 8B and 70B, both built on Llama-3. On the biomedical benchmark suite its own model card reports, the 70B is genuinely dominant, averaging 86.06 across nine tasks and beating Med-PaLM-2 and GPT-4, with 78.16 on MedQA. The 8B averages 72.50, with 58.99 on MedQA 4-option and 74.12 on PubMedQA, which is respectable for its size.

Now the asterisk, and it is a big one. An independent study deliberately chose benchmarks likely to fall outside the fine-tuning data of biomedical models, then compared them against their general-purpose counterparts. On NEJM clinical case challenges, OpenBioLLM-8B scored 30% against 64.3% for Llama-3-8B-Instruct. Not a marginal difference. A collapse. The 70B held up far better (66.4% versus 65% for Llama-3-70B-Instruct on JAMA cases), which tells you the fragility is concentrated in the small models, exactly the ones most of us can run.

Two practical warnings if you do try it. First, the model is genuinely sensitive to its prompt format: the authors specify the exact Llama-3 instruct chat template and a temperature of zero, and some community GGUF repositories ship without the correct template embedded, which means you can silently benchmark a crippled model. That is not a hypothetical. Check the template before you draw any conclusion about this model, because you can produce a spectacularly bad result purely by loading the wrong one, and you will have no idea. Second, the authors themselves advise against using it for direct patient care or clinical decision support, restricting it to research and exploration. Licence is the Llama 3 Community License, not something more permissive.

BioMistral: the permissive, multilingual outlier

BioMistral-7B takes Mistral-7B and continues pre-training on PubMed Central open-access text. It is smaller in ambition than the others and correspondingly modest in its claims, but it has two properties nothing else here matches.

It is Apache 2.0. In a landscape where MedGemma carries bespoke terms of use and everything Llama-derived carries the Llama community licence, a genuinely permissive licence is not a footnote, it is a strategic asset for anyone thinking about building something they might one day want to distribute.

And it is multilingual by design, covering eight languages including Italian, French, Spanish and German, with a 32K context. For clinicians working outside the anglosphere this is not a nice-to-have. Almost every medical benchmark in this article is in English, and almost every model here was tuned predominantly on English corpora, which means we know remarkably little about how any of them reason in Italian, or in Polish, or in Portuguese. A model that was built multilingual from the start deserves attention on that basis alone.

The honest counterweight: an evaluation of cancer communication found that BioMistral and Meditron exhibited higher toxicity and bias scores than general LLMs, and that medical models hallucinated more frequently than general ones, with Llama 3 showing the lowest hallucination rate. Domain fine-tuning appears to buy knowledge at the cost of safety and coherence, at least the way it has been done so far.

What actually fits on a 24 GB card

Here is the practical reality for anyone with a consumer workstation. On a 24 GB GPU, reserve two to four gigabytes for the KV cache and runtime overhead, which leaves roughly twenty to twenty-one gigabytes of usable space for weights at a modest context length.

Under that budget, almost everything in this article fits comfortably, and the flagship fits with room to spare:

The 4B and 7-8B models (MedGemma 1.5 4B, Meditron3-8B, OpenBioLLM-8B, BioMistral-7B) all sit entirely in VRAM even at generous quantisation levels. OpenBioLLM-8B, for instance, is 4.92 GB at Q4_K_M and 6.6 GB at Q6_K. You can afford Q6_K or Q8_0 here, which means you are not compromising the model to make it fit.

MedGemma 27B text-only at Q4_K_M is about 16.5 GB, which lands inside 24 GB with margin. This is the one that surprises people. The best-performing medical text model in the open ecosystem runs fully on a single consumer card. Push to Q5_K_M (18.8 GB) and you are still inside, though the context window you can afford starts to shrink, and that shrinkage bites sooner than the arithmetic suggests. It is the KV cache, not the weights, that quietly eats the last two gigabytes once you start feeding it a long document.

The 70B models (OpenBioLLM-70B, Meditron-70B, Apertus-70B-MeditronFO) are where the wall is. At Q4_K_M they need roughly 40 GB, so on a 24 GB card they spill onto system RAM and become CPU-bound. They will load. They will not be pleasant. Treat them as an overnight reference run rather than an interactive tool.

One practical aside, since it cost me an evening to work out. Not all runtimes behave the same way at the edge of VRAM. Ollama decides the GPU and CPU layer split for you, and does it conservatively, so a model that ought to fit ends up partly on the processor without announcing why it has suddenly slowed to walking pace. LM Studio exposes the offload manually. You can push a 27B model right up against the ceiling and know exactly what you did to it. Same underlying engine, very different experience once the margin gets thin.

A note on quantisation that people get wrong: dropping to Q4 typically costs a few percentage points of accuracy, which is usually acceptable, but the trade is not free, and comparing a Q2 medical model against a Q8 generalist and declaring a winner is not a comparison at all. Keep the quantisation constant when you compare, or you are measuring the wrong thing.

The summary table

Model	Base	Sizes	Licence	Quant for 24 GB	Footprint	Key published benchmark	Practical note
MedGemma 1.5 4B	Gemma 3 4B	4B multimodal	HAI-DEF terms of use	Q8_0	~4.5 GB	MedQA 69%; lab extraction F1 78%	Best small option; strong on structured extraction
MedGemma 27B (text)	Gemma 3 27B	27B text + 27B multimodal	HAI-DEF terms of use	Q4_K_M	~16.5 GB	MedQA 87.7%	Strongest open medical text model; fits a 24 GB card
Meditron3-8B	Llama 3.1 8B	8B (also 70B)	Llama community	Q6_K	~6.6 GB	Trained on 46k+ clinical guidelines	Independent evals unflattering; degrades under RAG
OpenBioLLM-8B	Llama 3 8B	8B, 70B	Llama 3 community	Q6_K	~6.6 GB	MedQA 58.99; 9-task avg 72.50	Chat template sensitive; collapses on out-of-distribution cases
BioMistral-7B	Mistral 7B	7B	Apache 2.0	Q6_K	~6 GB	Multilingual (8 languages), 32K context	Only permissive licence here; the multilingual choice
Apertus-8B / 70B-MeditronFO	Apertus (fully open)	8B, 70B	Fully open pipeline	Q6_K / offload	~6.6 GB / ~40 GB	70B avg 53.77 (best fully open)	Auditable data provenance; European bases

So do these models earn their place?

Read the evidence honestly and an uncomfortable pattern emerges. The medical fine-tunes reliably win on the benchmarks their creators report. They frequently lose on benchmarks their creators did not choose. The study that deliberately went looking outside the fine-tuning distribution concluded that fine-tuning LLMs on biomedical data may not provide the expected benefits and may actually reduce performance, which is a direct challenge to the premise the entire category rests on.

And the effect is size-dependent in a way that should worry anyone running local hardware. OpenBioLLM-70B roughly matched its generalist base on real clinical cases. OpenBioLLM-8B was cut in half by it. The Fully Open Meditron paper points at the same asymmetry from the other direction: smaller bases benefited most from medical fine-tuning, gaining up to 12.8 points, because the strong modern bases already contain so much of the medical knowledge that continued pre-training used to add. When the foundation is good enough, the speciality layer has less left to contribute, and can apparently do harm.

Note carefully what that does and does not mean. It does not mean the medical models are worthless. MedGemma 27B is, on the published evidence, the best open medical text model available and it fits on hardware you can buy. It means the label “medical” on a model is a claim, not a guarantee, and that it should be treated the way we treat any other claim in this profession: as something to be tested rather than accepted.

I am aware that this reads as more confident than the evidence strictly permits. The independent studies are few, the benchmarks are heterogeneous, and the field moves fast enough that a paper from eighteen months ago describes a landscape that no longer exists. What I can say is that the burden of proof has shifted. Two years ago the sensible default was to assume the medical fine-tune was better. It is no longer obvious that it is.

What a real evaluation would have to look like

Here is the uncomfortable part. Nothing in the published literature tells you how any of these models performs on your specialty, in your language, on the specific tasks you would actually delegate to it. The benchmarks are general medicine, overwhelmingly in English, and largely built from question banks and case archives that have been on the public internet for years, which is precisely the material these models were pre-trained on. Google says so about its own model, warning of data contamination and advising validation on datasets that are not publicly available.

So if you are considering putting one of these models to work, the evaluation has to be local, and it has to have a shape. At minimum:

Pair every medical model with its own base. Not with GPT-4, not with whatever is topping a leaderboard. With the exact generalist model it was fine-tuned from, at the same size and the same quantisation. Anything else measures scale or architecture, not the medical layer.

Write your own items. If your test questions can be found online, you are measuring memorisation, not competence. Items authored inside your department, about the cases you actually see, are the only ones you can trust.

Test more than knowledge. Multiple-choice accuracy is the easiest thing to measure and the least like clinical work. Add open reasoning. Add structured extraction, because pulling fields out of a discharge summary is the task most hospitals would actually want. And add false premises: ask a question built on a clinical error and see whether the model corrects you or agrees with you. The sycophancy literature suggests you will not like the answer.

Score blind. You will unconsciously favour the model you expect to win.

None of that is exotic. It is ordinary methodological hygiene, the same standard we would demand of any diagnostic test before letting it near a patient. The strange thing is how rarely it is applied to these models, given how confidently they are being recommended.

Key Takeaways

• MedGemma is the strongest option on published numbers: 87.7% MedQA for the 27B text model, and it fits on a 24 GB consumer GPU at Q4_K_M (~16.5 GB).
• MedGemma 1.5 (January 2026) updated the 4B model, lifting MedQA to 69% and substantially improving structured extraction from lab reports and EHR text.
• Meditron’s distinguishing feature was training on clinical guidelines, but independent evaluation found Meditron-70B scored lowest among 70B models on a neurology board exam and got worse, not better, under RAG.
• OpenBioLLM tops its own biomedical benchmark suite but collapsed on out-of-distribution NEJM cases (8B: 30% versus 64.3% for its Llama-3 base). It is also sensitive to the chat template, which can silently corrupt results.
• BioMistral-7B is the only genuinely permissive licence (Apache 2.0) and the only one built multilingual from the start, which matters enormously outside the anglosphere.
• The evidence increasingly suggests medical fine-tuning may add less than assumed, especially at small sizes where strong generalist bases already carry the knowledge.
• No published benchmark tells you how these models behave on your specialty, in your language, on your tasks. That evaluation has to be done locally, with items that are not on the internet.
• Every model here is explicitly not clinical grade. Their own authors say so. Use synthetic or de-identified data, and validate anything you intend to rely on.

Looking Ahead

The interesting question is no longer whether a hospital can run a capable model offline. It plainly can: the strongest open medical text model available fits on a single consumer graphics card, which would have sounded absurd two years ago.

The question is whether the domain-specific model is the right thing to run at all. The evidence is drifting, slowly and awkwardly, toward a conclusion the field has not fully absorbed: as the general-purpose bases get stronger, the medical layer on top has less and less left to add, and increasingly appears to subtract. If that trend continues, the future of clinical AI may look less like specialised medical models and more like excellent general models, carefully grounded in retrieval over verified sources, and evaluated locally against tasks that actually resemble the work.

Which would be a slightly deflating conclusion for a field that has invested heavily in the other idea. It would also be good news for anyone who wants to deploy this technology responsibly, because a general model with a transparent retrieval layer is far easier to audit than a black box with “Med” in its name.

For now, the honest position is that we do not know, and that the people best placed to find out are clinicians with a specialty, a GPU, and enough scepticism to test the claim rather than repeat it.

References

1. MedGemma model card – Google, Health AI Developer Foundations
2. MedGemma: our most capable open models for health AI development – Google Research
3. MedGemma Technical Report – Sellergren et al., arXiv:2507.05201
4. Meditron: an open-source suite of medical LLMs – EPFL LLM Team
5. MEDITRON-70B: Scaling Medical Pretraining for Large Language Models – Chen et al., arXiv:2311.16079
6. Fully Open Meditron: An Auditable Pipeline for Clinical LLMs – arXiv:2605.16215, 2026
7. Llama3-OpenBioLLM-8B model card and benchmarks – Saama AI Labs
8. BioMistral: A Collection of Open-Source Pretrained LLMs for Medical Domains – Labrak et al., arXiv:2402.10373
9. Biomedical Large Language Models Seem not to be Superior to Generalist Models on Unseen Medical Data – Bressem et al., arXiv:2408.13833
10. Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility – arXiv:2505.10472
11. General-purpose large language models outperform specialized clinical AI tools on medical benchmarks – Nature Medicine, 2026

Disclaimer: None of the models discussed here is approved as a medical device or validated for clinical use. Their developers state this explicitly. Nothing in this article constitutes clinical or regulatory advice.