Hunting for the largest language model you can run on a personal computer has turned into a discipline of its own. Forums, repositories and social feeds circulate configurations that load models of 70, 120, even several hundred billion parameters, using consumer GPUs, generous amounts of system RAM and increasingly aggressive quantisation.
The results are frequently impressive as engineering. A model that would need well over a hundred gigabytes at FP16 precision can be compressed, spread across GPU and system memory, and finally brought up on a workstation that is not out of reach.
There remains a less spectacular question, and a far more consequential one:
If the model produces one or two tokens per second, can we honestly call it usable?
Being able to load a model and being able to work with it are two different achievements. The first is a memory problem. The second involves latency, bandwidth, quality, context, power draw and human time.
The distinction matters especially in healthcare. Local processing can offer real advantages for certain workflows — those touching sensitive data, operational continuity, or integration with internal systems. But precisely because the setting is critical, a solution cannot be judged by the number of parameters it manages to load. It has to be reliable, governable, and fast enough to slot into workflows as they actually run.
“It runs locally” is not a benchmark
When someone describes a local installation, the phrase “the model runs” can mean any of several quite different things:
- the process started without exhausting memory;
- the model produced at least one reply;
- generation happens at an acceptable speed;
- the system stays fast with long documents;
- quality is sufficient for the intended task;
- several users can work with it at once;
- the output arrives within the operational window required.
Demonstrations published online usually verify only the first two.
Hence a misunderstanding: getting the model to start is treated as proof that it is useful. But an interactive system is not judged on whether it eventually completes a computation. It is judged on how long it takes to hand back something you can use.
Assessing a local LLM takes at least four measurements.
Time to first token, or TTFT. How long passes between sending the request and the start of the reply.
Prompt processing. How quickly the system chews through the incoming text. This becomes decisive when the prompt carries a clinical record, a document, a transcript, or a set of passages retrieved through a RAG pipeline.
Generation speed. The number of tokens produced per second during the reply.
Time to complete response. The metric closest to what the user perceives, and to the operational value of the system.
A model can generate at a decent clip yet take a long time to digest a lengthy document. Or it can start replying almost immediately and then spend several minutes finishing a long chain of reasoning. Tokens per second alone do not tell the whole story — but ignoring them means overlooking one of the main reasons local deployments end up unusable.
Why generation is so sensitive to memory
Inference in an autoregressive model has two main phases.
During prefill, the model processes the prompt and builds the representations it needs to begin answering. This work parallelises well and is typically limited by compute capacity.
During decode, tokens come out one at a time. Each new token depends on the ones before it. In this phase, how fast weights, cache and activations can be read from memory usually matters more than the GPU’s theoretical arithmetic throughput. Interactive generation, in other words, tends to be a memory-bound workload [1].
That explains why VRAM is not merely a place to park the model. It is also very high bandwidth memory.
When a substantial portion of the model is moved into system RAM, the capacity problem may be solved, but the system now has to route part of the work through a far slower memory subsystem. It is not enough for the model to “fit” into the sum of RAM and VRAM: what counts is where its components sit during each generation step.
Quantisation: less memory, but not for free
The most widespread technique for running large LLMs locally is quantisation. Weights, normally held as 16-bit numbers or larger, are converted into reduced-precision formats.
As a first approximation, a 70-billion-parameter model needs roughly 140 GB for weights alone at FP16. Quantising to 8 bits can halve that footprint; 4-bit can cut it to about a quarter, before you add metadata, buffers and cache on top [2].
Four-bit quantisation is often held up as the sweet spot. A good deal of published work shows that, with the right techniques, it preserves much of the original model’s quality. That observation does not license treating every format or every compression level as equivalent.
As you push down towards 3 or 2 bits:
- the risk of distorting the model’s output distributions rises;
- some capabilities degrade unevenly;
- available context may shrink;
- outcomes depend heavily on calibration and on the format chosen;
- apparent coherence and actual quality can drift apart.
The llama.cpp project uses perplexity, KL divergence, probability shifts and top-token agreement rate to quantify the loss introduced by quantisation [3]. The point deserves emphasis: confirming that a model still produces grammatically plausible text says nothing about whether it retains the original’s capabilities.
Quantisation also affects speed in ways that are not uniform. A more compact format reduces the volume of data to be read, but it demands kernels and runtimes able to process it efficiently. Compression that looks advantageous on paper does not automatically translate into a speed-up on arbitrary hardware.
ExLlamaV2, to take one example, showed that a Llama 2 70B at roughly 2.5 bits can run on a 24 GB GPU at speeds in the tens of tokens per second [4]. A notable result — but obtained with extreme quantisation, an optimised runtime and a constrained context. It is not a general rule for any 70B, nor does it show that quality and performance match the original.
So the right question is not “how far can I compress this model?” but:
What is the highest level of compression compatible with the quality my use case demands?
In healthcare that question has to be asked task by task. A quantisation that works perfectly well for sorting administrative documents may prove inadequate for pulling out rare clinical findings or interpreting complex relationships.
CPU offload: the model fits, the speed leaves
When VRAM runs short, runtimes such as llama.cpp let you keep some layers on the GPU and the rest in system RAM [5]. This hybrid CPU–GPU inference makes it possible to run models that the available hardware could not otherwise host.
It is also the technique that exposes the gap between feasibility and usability most plainly.
Discussions in the LocalLLaMA community contain plenty of examples of 70B models running on a single 24 GB GPU with the remaining weights in RAM. Reported performance varies with the model, the quantisation, the processor, the memory, the context and the number of layers pinned to the GPU.
In one case, a system with an RTX 4090 and 64 GB of DDR5 generated around 1.2 tokens per second on a 70B Q5 partially resident in RAM. In the same thread, a far more aggressive quantisation — small enough to sit almost entirely on the GPU — reached roughly 10 tokens per second [6].
These figures are anecdotal and should not be read as scientific benchmarks. They are useful all the same for describing the phenomenon: having a very fast GPU does not prevent the slowdown once a meaningful share of the processing falls back on system memory.
Another test, with a 14B model, recorded around 70 tokens per second as long as model and context stayed inside VRAM. Once capacity was exceeded, RAM and CPU stepped in and generation dropped to about 19 tokens per second — even though most of the load was still nominally assigned to the GPU [7].
Offload remains valuable for:
- batch processes with no interactive requirement;
- experimentation and occasional evaluation;
- overnight generation runs;
- environments where capacity matters more than latency;
- models used rarely, as an escalation tier.
It is far less convincing for chatbots, voice assistants, code completion, and clinical workflows where a professional has to wait for the output before carrying on.
Context consumes memory and time
Weights are not the only thing occupying memory. During inference the system maintains a KV cache holding information about tokens already processed. Its size grows with context length, batch size and model architecture.
NVIDIA estimates that for Llama 3 70B, a 128,000-token window can require around 40 GB of KV cache for a single user [8].
The theoretical ability to accept very long contexts is therefore no guarantee that those contexts are usable on the system in front of you. As context grows, you may see:
- less VRAM left for the weights;
- part of the load shifting into RAM;
- longer time to first token;
- attention slowing down;
- fewer concurrent users supported;
- out-of-memory failures.
KV cache quantisation, Flash Attention, prompt caching, sliding windows and selective context reduction can all take the edge off. But the most effective improvement is usually the least glamorous one: stop sending the model information it does not need.
For a healthcare application, that means designing extraction, retrieval and document segmentation with care, rather than leaning indiscriminately on ever-larger context windows.
How many tokens per second do you actually need?
There is no universal threshold. A batch procedure can be useful at one token per second; a voice assistant can feel broken at ten. For a single user, though, a rough scale is workable.
| Generation speed | Indicative experience |
|---|---|
| Under 2 tokens/s | Technical demo or batch process |
| 2–5 tokens/s | Occasional use with obvious waiting |
| 5–10 tokens/s | Slow but workable interaction |
| 10–20 tokens/s | Conversation generally practicable |
| 20–50 tokens/s | Fluid experience |
| Above 50 tokens/s | Perceived as very fast |
NVIDIA cites 5 tokens per second as an example below typical reading speed, and 50 tokens per second as an example of an excellent experience [9]. These are indications, not clinical or universal thresholds.
To see what the number means in practice, take a 1,000-token generation:
- at 2 tokens/s it takes about 8 minutes 20 seconds;
- at 5 tokens/s, about 3 minutes 20 seconds;
- at 10 tokens/s, about 1 minute 40 seconds;
- at 50 tokens/s, about 20 seconds.
With reasoning models the problem can be sharper still. If the system emits thousands of intermediate tokens, a speed that would be acceptable for a short answer can produce latencies of several minutes.
In a healthcare workflow the effect is not merely perceptual. Excessive latency can push the user to abandon the tool, run tasks in parallel, lose the operational thread, or fall back on alternative procedures that nobody is governing.
Unified memory, multiple GPUs and speculative decoding
Other strategies exist for getting past the single-GPU ceiling.
Unified memory systems let CPU and GPU draw on a common pool. This softens the rigid split between RAM and VRAM and makes larger models loadable. It does not remove the bandwidth constraint: capacity grows, but speed still depends on architecture and runtime.
Using multiple GPUs allows weights and compute to be distributed. The benefit hinges on how fast the cards can talk to each other. Technologies such as NVLink and NVSwitch exist precisely so the interconnect does not become the new bottleneck [9]. Two consumer GPUs joined only over PCIe are not automatically equivalent to a single GPU with the same total memory.
Speculative decoding takes a different route: a smaller model proposes several tokens, which the main model then verifies in a block. When proposals are accepted often enough, the technique speeds up generation without altering the target model’s distribution. llama.cpp supports both draft models and n-gram-based methods [10].
A promising optimisation, then, though not a universal answer. It costs additional memory for the draft model, and it is unlikely to turn an offload-dominated configuration into a genuinely fast one.
The hidden cost of “free and local”
Local execution is often set against cloud APIs on the basis of per-token cost alone. That comparison is incomplete.
A large local model also brings:
- hardware purchase and depreciation;
- energy consumption;
- cooling and noise;
- maintenance of drivers, runtimes and quantisations;
- time spent on testing and updates;
- slower iteration;
- unused capacity during idle periods;
- the cost of the time spent waiting for answers.
In a healthcare organisation, governance costs join the list:
- access control;
- environment segregation;
- logging and audit;
- version management;
- performance verification;
- quality assessment;
- error monitoring;
- operational continuity.
Local does not automatically mean secure, compliant or governed. It means the organisation retains greater control over the execution environment. That control still has to be exercised through appropriate architecture and procedures.
Why local still matters in healthcare
Health data falls within the categories of personal data given special protection under the European framework. The European Data Protection Board has also reiterated that the development and use of AI models must be assessed against GDPR principles, including effective anonymity, lawful basis and the processing of personal data [11].
The European Health Data Space Regulation, for its part, builds a framework for access, control, exchange and secure reuse of electronic health data. It entered into force on 26 March 2025, with its provisions applying progressively [12].
None of this implies that every piece of healthcare processing has to happen locally. Compliance depends on purpose, lawful basis, roles, contracts, technical measures, localisation, minimisation and a good many other factors.
There are nevertheless cases where processing kept local, or inside the organisation’s own infrastructure, offers concrete advantages:
- classifying documents containing identifying data;
- extracting fields from reports and forms;
- semantic indexing of internal archives;
- preliminary anonymisation or pseudonymisation;
- searching confidential procedures and documentation;
- transcription and normalisation within authorised environments;
- activities that must continue without external connectivity;
- applications where sending content to a remote service is not contemplated by the governance model.
For tasks of this kind, a small, fast, specialised model can deliver more value than a very large but sluggish one.
Parameter count should not stand in for validation. Within a well-bounded process, a smaller model can be evaluated against a representative dataset, constrained to a structured output format, and wrapped in a pipeline with deterministic checks.
Hybrid architecture as a design choice
The opposition between “all local” and “all cloud” is often artificial. A hybrid solution can assign each activity to whichever environment suits it.
One possible flow:
- local processing of identifying data;
- classification of the request with a compact model;
- local execution of simple or sensitive tasks;
- removal or transformation of unnecessary data;
- controlled escalation to a more capable model when the task warrants it;
- local validation and logging of the outcome.
The local model can handle:
- routing;
- structured extraction;
- normalisation;
- short summaries;
- document search;
- template application;
- preliminary checks;
- protection or removal of identifiers.
The remote or centralised model can be reserved for:
- complex reasoning;
- exceptionally long documents;
- ambiguous requests;
- infrequent problems;
- second-line verification;
- tasks where the local model does not reach the required confidence.
Escalation should not fire automatically and indiscriminately. It needs to be governed through rules, data classification, consent or authorisation where required, content minimisation, and a choice of providers and configurations consistent with the organisational framework.
Seen this way, the local model is not an impoverished version of the cloud model. It is a different component, tuned for privacy, latency, control and specialisation. The larger model becomes a resource to be drawn on selectively, not the obligatory engine behind every request.
From model-first to task-first
The race to run the biggest possible LLM locally usually starts from a model-first posture: pick the model, then look for a way to bend the hardware around it.
A useful architecture should start from the task:
- What accuracy is required?
- What latency is acceptable?
- What data may the model see?
- How much context is really needed?
- Is the process interactive or batch?
- What happens when the model gets it wrong?
- Is there a human in the loop?
- When is escalation permitted?
- What level of availability is required?
Only after these questions does it make sense to choose size, quantisation, runtime and placement.
The headline metric is not the maximum number of parameters you can load. It is the smallest system capable of completing the task correctly within the constraints you have set.
Conclusion: better small and functional
Running a model with tens of billions of parameters locally, on consumer hardware, is a fascinating technical achievement. Quantisation, offload, unified memory, multi-GPU distribution and speculative decoding have vastly expanded what can be done outside the data centre.
Technical possibility is not the same as usefulness.
A model generating one or two tokens per second may suit a batch process, an experiment, or a task you run rarely. It should not be presented as an interactive solution merely because it manages to produce an answer.
In healthcare, local retains strategic value. It can reduce data exposure, support controlled processes and provide operational independence. For that value to materialise, though, the system has to be fast enough, validated on the task, and embedded in adequate governance.
Real success is not seeing a 70- or 120-billion-parameter model appear in the process list. It is getting an answer good enough, quickly enough, that you can carry on working.
Better small and functional than large and unusable.
And when the task genuinely calls for more capability, the rational answer is not to wait longer: it is to adopt a hybrid architecture.
References
- NVIDIA, “Mastering LLM Techniques: Inference Optimization”, 2023. Online.
- Hugging Face, “Bitsandbytes — Quantization”, Transformers documentation. Online.
- ggml-org, “Perplexity and quantization quality metrics”, llama.cpp. Online.
- turboderp-org, “ExLlamaV2: performance and EXL2 quantization”. Online.
- ggml-org, “llama.cpp: LLM inference in C/C++”. Online.
- LocalLLaMA, “Llama 3 70B Instruct works surprisingly well on 24GB VRAM cards”, community discussion. Online.
- Windows Central, “Just what sort of GPU do you need to run local AI with Ollama?”, 2025. Online.
- NVIDIA, “Accelerate Large-Scale LLM Inference and KV Cache Offload with CPU-GPU Memory Sharing”, 2025. Online.
- NVIDIA, “NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Language Model Inference”, 2024. Online.
- ggml-org, “Speculative Decoding”, llama.cpp documentation. Online.
- European Data Protection Board, “EDPB opinion on AI models: GDPR principles support responsible AI”, 18 December 2024. Online.
- European Commission, “European Health Data Space Regulation”. Online.