Introduction
For a few years now, “artificial intelligence” has mostly meant large language models. The routine is familiar: we type a prompt, the model returns a stream of words, and someone reads it (or some other piece of software parses it).
Jev starts somewhere else.
TypeSafe AI released it in September 2026 as its first System One Model. It isn’t really meant to chat, explain or write. Its job is to make structured probabilistic decisions that software can consume directly; TypeSafe’s own shorthand is unstructured state in, typed probabilistic decisions out.
On paper, a minor architectural detail. Once you actually use it, it changes how you call the model and, more to the point, how you should read what comes back.
My interest was medicine, where a good share of the work has little to do with producing text. Clinicians spend much of the day choosing between alternatives, asking whether a hypothesis is supported well enough, weighing evidence and, usually without saying so, deciding when things are too uncertain to let an automated system act.
So I ran a short series of experiments. Controlled clinical scenarios first. Then 2,040 real medical questions that I had already used to benchmark a group of local language models.
I wasn’t trying to establish clinical safety; that would take a completely different study. I wanted to understand what this class of model actually does, how stable its decisions are, and how it copes with medical reasoning.
1. What is Jev, and why is it different?
A conventional LLM generates autoregressively: one token at a time, each conditioned on what came before. Ask it for JSON, a class label or a single number and it is still, underneath, writing text.
Jev gives that up on purpose.
In TypeSafe’s framing, System One Models are meant to make decisions inside software. You don’t let the model produce an arbitrary answer and then validate or parse it afterwards. You declare the output structure up front, and the model returns typed values with probabilities and confidence estimates attached.
The company also describes a different training objective, Reinforcement Learning for Calibrated Decisions (RLCD), and a parallel sampling architecture meant to make structured decisions faster and cheaper than an equivalent generative pipeline. For now these are the developer’s claims. I had no way to verify them, and nobody has yet shown that they hold across domains.
The conceptual distinction, though, deserves to be taken seriously.
Some questions are a natural fit for a generative model:
- “Explain this ECG.”
- “Summarize this paper.”
- “Write a discharge letter.”
- “Generate Python code.”
Others suit a decision model better:
- “Which category does this case belong to?”
- “Is this proposition sufficiently supported?”
- “How strong is the evidence?”
- “Should this case be escalated for human review?”
None of this makes Jev a substitute for an LLM. It is a different component, with a different job.
One practical point runs through the whole article, so I’ll state it now. Jev is a proprietary service, accessed through a remote API. The language models I compare it with later ran locally on my workstation, often in quantized form. What I’m comparing is performance on the same tasks. Not deployment conditions, not privacy, not hardware requirements or computational efficiency.
Figure 1. One clinical state can be evaluated through three different Jev primitives: Choice, Noul, and Score. These outputs answer different questions and should not be interpreted as interchangeable probabilities.
2. How should Jev’s answers be interpreted?
The numbers Jev returns are easy to misread. That was the first thing I learned, and probably the one I’d stress most.
I worked with three primitives: Choice, Noul and Score.
To see how they differ, I built a simple scenario.
A 67-year-old woman presents with sudden pleuritic chest pain and dyspnea, ten days after a knee replacement. She is tachycardic and hypoxemic, with mild swelling of one calf. The ECG shows sinus tachycardia, the chest X-ray is unremarkable, and no definitive test for pulmonary embolism has been done yet.
Same clinical state. Three different questions.
Choice
With Choice, Jev had to pick the best-supported diagnosis among pulmonary embolism, acute coronary syndrome, acute aortic syndrome, pneumonia, or insufficient information.
It picked pulmonary embolism, with a confidence of 1.00.
The probability distribution that comes with it expresses a preference among the options offered in that particular question. Reading it as the posterior probability that this patient actually has a PE would be a mistake: change the list of options and, in principle, the probability mass gets redistributed.
Noul
Here the question was a different one: does the available information give substantial reason to suspect acute pulmonary embolism?
Answer: 0.94.
That is a judgment about how well a proposition is supported. It isn’t the same quantity as the Choice probability and, without calibration on domain-specific data, I wouldn’t convert it into a clinical risk percentage.
Score
Last, I defined an ordered scale for the strength of evidence:
- little or no support
- weak support
- moderate support
- strong support
- very strong support
Jev returned 3.18, with most of the probability distribution sitting around “strong”.
The three answers agree with one another. They just aren’t measuring the same thing.
Once a model like Jev is embedded in software, this stops being an academic point. One question might select the diagnostic category, another decide whether an alert fires, a third whether a clinician gets called. Confusing those numbers would be an easy bug to write, and not a harmless one.
3. Does Jev respond coherently to changing evidence?
Before scaling up to a larger benchmark, I wanted two basic checks.
Controlled perturbation
In the first one I changed the strength of the evidence and left the questions alone.
Three versions of the pulmonary embolism scenario:
- reduced evidence: recent surgery, immobilization, previous venous thromboembolism, cancer and unilateral leg swelling all removed;
- baseline: recent knee replacement and unilateral calf swelling;
- strengthened evidence: CT pulmonary angiography explicitly showing acute bilateral segmental pulmonary emboli.
Jev moved the way one would expect.
Noul: 0.79 → 0.94 → 0.98
Evidence Score: 2.23 → 3.18 → 4.00
Choice confidence rose as well. (A Noul of 0.79 in the reduced version is not unreasonable, by the way: pleuritic pain, dyspnea, tachycardia and hypoxemia are still there.)
This says nothing about clinical accuracy or calibration. What it shows is more elementary: when I deliberately strengthened the relevant evidence, Jev’s decision signals moved together, and in the right direction.
Figure 2. Controlled perturbation experiment. Noul, normalized Score, and Choice confidence all increased as the evidence for pulmonary embolism was strengthened.
Repeatability
Then I sent each of the three clinical states 15 times, without touching the wording, questions, options or scoring criteria.
Choice returned the same answer in all 45 calls.
In the baseline scenario, mean Noul was 0.9440 ± 0.0051 and mean Score 3.1907 ± 0.0294. The strengthened-evidence version barely moved at all.
Good news, within limits. Repeatability is not truth, and a model can be wrong very consistently. All this showed was that the outputs were stable enough to justify a larger evaluation.
4. Jev on real medical questions
For the main test I went back to the dataset I had put together for an earlier benchmark of medical and general-purpose LLMs.
The advantage was obvious. Jev could be scored on exactly the same question IDs the local models had already answered, so the comparison could be done item by item.
The analysis covered 2,040 questions from:
- MedQA
- MedMCQA
- medical subsets of MMLU
- PubMedQA
- MedExpQA, including both English and Italian items
I left out MedCalc-Bench. Many of its answers are numerical or free-form, and they don’t map cleanly onto the Choice primitive.
Out of the 2,040 compatible questions, Jev answered 1,729 correctly: an overall accuracy of 84.75%.
Comparison with the local models
In the earlier benchmark the two strongest local models were Qwen3.8-27B and Muse-Glimmer-30B.
On the same 2,040 items, the overall accuracies were:
| Model | Accuracy |
|---|---|
| Jev | 84.75% |
| Qwen3.8-27B | 83.33% |
| Muse-Glimmer-30B | 83.04% |
| MedGemma-27B-Text-IT | 75.59% |
| Llama3.1-Aloe-Beta-8B | 66.57% |
| Meta-Llama-3.1-8B-Instruct | 64.36% |
| MediPhi-Instruct-3.8B | 59.02% |
| Phi-3.5-mini-instruct-3.8B | 58.09% |
Since the items were identical, a paired analysis was possible.
Against Qwen3.8-27B, Jev was right on 128 items that Qwen got wrong, and Qwen was right on 99 that Jev missed. After Holm correction the difference was not statistically significant (p = 0.0629).
Against Muse-Glimmer-30B the discordant counts were 130 and 95, with a Holm-adjusted p = 0.0464. Borderline, in other words.
With the other models the gaps were much wider.
So the fair reading is not that Jev crushed every LLM. Its performance sits in the same band as the strongest local models in this benchmark.
I actually find that more interesting than a clean win: Jev is not a generative language model, and it gets there through a quite different computational route.
Once more, though, this is a task-level comparison. Jev runs as a remote proprietary API; the other models ran locally, under hardware and quantization constraints I know well. Nothing here says anything about cost, privacy, reproducibility, energy use or flexibility of deployment.
Figure 3. Accuracy on the same 2,040 medical questions. The comparison is at task level only: Jev was accessed remotely through an API, whereas the comparator LLMs were run locally.
Confidence and selective prediction
From a practical standpoint, what I liked most is that confidence is part of the native output.
If that confidence carries real information, accuracy should climb as you keep only the decisions above progressively higher thresholds.
It did.
| Minimum confidence | Coverage | Accuracy among retained answers |
|---|---|---|
| 0.50 | 87.5% | 90.6% |
| 0.70 | 79.9% | 93.1% |
| 0.80 | 75.2% | 93.9% |
| 0.90 | 68.4% | 95.1% |
| 0.95 | 62.2% | 95.4% |
At 0.90, for instance, Jev kept about two thirds of the questions, and accuracy on those rose to roughly 95%.
That points towards selective prediction: software acts on its own only above a predefined threshold, and everything else goes to a human reviewer or to a second model.
The same table, however, also shows why confidence can’t be treated as a safety guarantee. Even at ≥0.90 there were still 69 disagreements with the benchmark reference.
Figure 4. Selective prediction. Increasing the minimum confidence threshold improved accuracy among retained answers, but progressively reduced coverage.
When the benchmark itself is uncertain
I went through those 69 disagreements by hand.
It turned out to be more informative than I had expected.
After a two-stage post-hoc review:
- 51 looked like likely mismatches in the benchmark or in its reference label
- 10 were clearer Jev errors
- 8 stayed ambiguous, or depended on nuances of the source
These numbers must not be used to raise Jev’s official accuracy after the fact.
The review was post hoc, unblinded, and triggered by Jev’s own disagreements. More importantly, I did not carry out the same review on the errors of the other seven models. “Correcting” Jev’s score while leaving theirs untouched would simply be bad methodology.
The audit is useful for another reason: it exposes a problem that affects medical AI benchmarking in general.
Some questions really do admit more than one defensible answer. Some reference labels seem inconsistent with their own source. PubMedQA is a good example. Nuanced conclusions such as may be involved, could be useful in selected patients or the evidence remains uncertain eventually have to be squeezed into yes, no or maybe, and something gets lost in that compression.
A model can therefore disagree with a benchmark without being medically wrong.
The review also turned up genuine high-confidence failures, though. In some cases Jev took a categorical position where the source had deliberately left the question open; in others it simply chose a clinically inappropriate answer.
If I had to keep a single practical lesson from the whole experiment, it would be this one:
confidence helps you tell which decisions are more reliable. It doesn’t certify that they are correct.
5. Where could a model like Jev be useful in medicine?
Nothing in these experiments shows that Jev is ready for clinical deployment.
Multiple-choice questions are not patients. I didn’t evaluate prospective outcomes, workflow safety, adversarial robustness, distribution shift, or behavior on live electronic health record data.
Still, the paradigm suggests a few applications worth thinking about.
Structured triage and routing
Plenty of healthcare workflows need a category rather than a paragraph: assigning a priority, picking the most appropriate specialty, flagging which documents need review, routing a message to the right clinical team.
These are decision problems by nature.
Selective decision support
A complex workflow could be broken down into several narrow decisions, with confidence thresholds determining where automation is allowed. Confident cases are processed automatically; uncertain ones are escalated to a clinician.
The threshold belongs to the workflow. It isn’t a universal number handed over by the model.
Classification of clinical or scientific text
Reviewing clinical notes, abstracts, incident reports or registry records at scale usually means extracting categories or decisions, not writing prose. When the possible outputs are known in advance, a structured model starts to look attractive.
Verification and model-to-model workflows
This is the possibility I find most intriguing: maybe Jev and generative LLMs aren’t competitors at all.
A decision model could take care of the structured judgments:
classify → score → verify → decide whether to escalate
A generative model could then handle communication:
explain → summarize → write
The resulting architecture might look something like:
structured decision model → generative model → human oversight
In medicine this separation has a certain appeal. The component that makes a decision doesn’t need to be the one that explains it, and keeping the two apart could make each of them easier to audit.
Conclusions
What makes Jev interesting is that it questions an assumption most of us stopped noticing during the rise of generative AI: that an intelligent model must, by necessity, produce language.
In these exploratory experiments, Jev responded coherently when I modified the clinical evidence, gave near-identical outputs across repeated calls, and was competitive on more than two thousand medical questions. On the same items, its accuracy was close to that of the strongest local LLMs I had tested, despite belonging to a very different class of model.
Its native confidence signal was informative as well, with higher-confidence subsets progressively more accurate. High-confidence failures remained, though. Uncertainty estimates should support oversight, not replace it.
The manual review of disagreements added one more lesson. Evaluating a medical AI system also means evaluating the benchmark you use to judge it, whether you intend to or not.
Jev does not show that decision models should replace LLMs.
What it suggests is subtler: for some tasks, generating language may not be the only way to use artificial intelligence, and perhaps not the most natural one.
Frequently asked questions
Is Jev an LLM?
Not in the usual sense.
TypeSafe describes Jev as its first System One Model: a model that receives a state and a set of structured questions, and returns typed probabilistic decisions rather than free generated text.
Does a confidence of 0.95 mean that a diagnosis is 95% likely to be correct?
No.
Neither the confidence signal nor the probability attached to a Choice output should be read, by default, as a calibrated posterior clinical probability.
In this experiment, higher confidence did go with higher accuracy. High-confidence errors still happened.
Is Jev a local model?
No.
Jev is currently accessed through TypeSafe’s hosted API, and that sets it clearly apart from the comparator models in this experiment, all of which ran locally.
Did Jev outperform the local LLMs?
It had the highest raw overall accuracy on the 2,040 shared questions.
The margin over Qwen3.8-27B and Muse-Glimmer-30B was small, though. The paired difference from Qwen was not statistically significant after Holm correction; the one from Muse-Glimmer was borderline.
And since Jev and the local models run under very different deployment conditions, the result is better read as a task-level comparison than as a general ranking.
Can Jev make confident mistakes?
Yes.
With a minimum confidence threshold of 0.90, accuracy among retained answers was about 95%, but disagreements with the benchmark were still there.
Confidence is a useful uncertainty signal. It is not a guarantee.
Why did some apparent errors turn out to be disagreements with the benchmark?
Because medical benchmarks are imperfect reference standards.
Some questions are ambiguous, some of the original scientific conclusions are nuanced, and some stored labels are open to question.
Our manual review covered only Jev’s high-confidence disagreements. The comparator LLMs’ errors were not adjudicated in the same way, so this analysis can’t be used to selectively correct Jev’s comparative score.
Is Jev ready for clinical use?
This experiment does not establish clinical safety, nor suitability for patient care.
The evaluation relied on medical question-answering tasks and controlled scenarios. Real clinical deployment would need prospective validation, a safety analysis specific to each workflow, proper governance, and human oversight.
Where can I try Jev?
Jev is currently available from TypeSafe AI in early access.
The official TypeSafe website is the place to start, and to join the waitlist:
TypeSafe AI — Jev and System One Models
If you already have access, the TypeSafe console is here:
For more technical background, see the official introduction to Jev:
Introducing System One Models and Jev — TypeSafe AI
Methodological note
This article reports an independent exploratory evaluation carried out on September 18, 2026, with Jev 1.13.0.
The medical benchmark consisted of 2,040 multiple-choice items compatible with Jev’s Choice primitive, all previously evaluated with seven locally deployed language models.
Every comparative accuracy analysis used the original benchmark reference labels.
The later review of high-confidence disagreements was exploratory and post hoc; it did not change the primary benchmark scores.
The experiments described here should not be taken as evidence of clinical efficacy or safety.