Part 2 of the “Local AI” series — Benchmarks and recommendations for CPU-only inference
In Part 1, we installed Ollama and ran our first local model. Now the question becomes: which model should you actually use?
The answer depends on your hardware and your patience. On CPU-only systems, model size directly impacts speed — and the difference between a 2B and an 8B parameter model isn’t just “slightly slower.” It can mean the difference between a 2-second response and a 40-second wait.
This article presents real benchmark data from our test system: an Intel i7-1165G7 with 20 GB RAM and no dedicated GPU. Every number here is reproducible on similar hardware.
The Test Methodology
All benchmarks were run using Ollama’s
--verboseflag, which provides precise timing data:
ollama run gemma2:2b --verbose
Test prompt: “Hello, tell me who you are in one sentence”
This simple prompt lets us measure raw generation speed without the complexity of long outputs or specialized tasks. We’ll test complex prompts in Part 3.
Key metrics: – eval rate (tok/s): Generation speed in tokens per second — the most important number – total duration: End-to-end time including model loading – eval count: Number of tokens generated
Benchmark Results
gemma2:2b — The Speed Champion
total duration: 2.0836445s load duration: 572.8004ms prompt eval rate: 39.14 tokens/s eval count: 15 token(s) eval duration: 1.0106817s eval rate: 14.84 tokens/s
| Metric | Value |
|---|---|
| Total time | 2.08 seconds |
| Generation speed | 14.84 tok/s |
| Tokens generated | 15 |
| Model size | 1.7 GB |
Response: “I am Gemma, an open-weights AI assistant developed by Google DeepMind.”
This is our baseline — and it’s excellent. Under 2 seconds for a complete response, with generation speed fast enough to feel interactive. For most daily tasks, gemma2:2b is the right choice on this hardware.
phi3:mini — The Verbose Alternative
total duration: 4.2624698s load duration: 59.172ms prompt eval rate: 23.63 tokens/s eval count: 39 token(s) eval duration: 3.2560085s eval rate: 11.98 tokens/s
| Metric | Value |
|---|---|
| Total time | 4.26 seconds |
| Generation speed | 11.98 tok/s |
| Tokens generated | 39 |
| Model size | 2.2 GB |
Response: “I’m approximating as an AI language model; my purpose is to help you with various computing tasks.”
Microsoft’s Phi-3 Mini is slightly slower but produces longer responses. It generated 39 tokens versus Gemma’s 15 for the same prompt. Whether this verbosity is helpful depends on your use case.
llama3.2 — The Balanced Option
total duration: 6.3287392s load duration: 337.6094ms prompt eval rate: 28.84 tokens/s eval count: 57 token(s) eval duration: 4.5909959s eval rate: 12.42 tokens/s
| Metric | Value |
|---|---|
| Total time | 6.33 seconds |
| Generation speed | 12.42 tok/s |
| Tokens generated | 57 |
| Model size | 2.0 GB |
Response: “Hello! I’m an artificial intelligence model known as Llama, created to help you find the information you’re looking for…”
Meta’s Llama 3.2 (3B) sits in the middle ground. Reasonable speed, good general capability. A solid choice if you want something different from Gemma.
llama3 (8B) — The Patience Test
total duration: 10.4371129s load duration: 1.0263656s prompt eval rate: 11.39 tokens/s eval count: 42 token(s) eval duration: 7.5762463s eval rate: 5.54 tokens/s
| Metric | Value |
|---|---|
| Total time | 10.44 seconds |
| Generation speed | 5.54 tok/s |
| Tokens generated | 42 |
| Model size | 4.7 GB |
Response: “Hello, I’m an artificial intelligence in English and Italian called LLaMA, created to help and entertain through conversations!”
Here’s where CPU-only inference starts to feel painful. At 5.54 tok/s, you’re watching words appear one… by… one. The model is more capable than the 2-3B options, but the speed penalty is severe. On this hardware, 8B models are usable for batch tasks but frustrating for interactive use.
gemma4:e4b — The Thinker
total duration: 37.437441s load duration: 721.2214ms prompt eval rate: 16.60 tokens/s eval count: 278 token(s) eval duration: 34.817568s eval rate: 7.98 tokens/s
| Metric | Value |
|---|---|
| Total time | 37.44 seconds |
| Generation speed | 7.98 tok/s |
| Tokens generated | 278 (including thinking) |
| Model size | 3.0 GB |
This one is different. Gemma 4 E4B includes visible chain-of-thought reasoning — it shows you how it thinks before answering.
Thinking process (actual output):
Thinking… 1. Analyze the Request: The user wrote “hello, tell me who you are in one sentence” 2. Determine Identity (Core Persona): – Name: Gemma 4 – Developer: Google DeepMind – Nature: Large Language Model (LLM) – Type: Open weights model 3. Constraint: The answer must be in *one sentence* 4. Drafting the Sentence: – Draft 1: I am Gemma 4, a large open-weights language model developed by Google DeepMind. 5. Final Polish and Selection 6. Final Output Generation …done thinking.
Final response: “I am Gemma 4, a Large Language Model with open weights developed by Google DeepMind, here to assist you and answer your questions.”
The 37-second wait feels justified when you can see the reasoning. This model is fascinating for educational purposes and complex tasks where you want transparency. But it’s not for quick queries.
Summary Table
| Model | Parameters | Size | Speed (tok/s) | Time | Use Case |
|---|---|---|---|---|---|
| gemma2:2b | 2B | 1.7 GB | 14.84 | 2.1s | Daily driver |
| phi3:mini | 3.8B | 2.2 GB | 11.98 | 4.3s | Detailed responses |
| llama3.2 | 3B | 2.0 GB | 12.42 | 6.3s | General chat |
| gemma4:e4b | 4B | 3.0 GB | 7.98 | 37.4s | Complex reasoning |
| llama3 | 8B | 4.7 GB | 5.54 | 10.4s | Batch only |
The pattern is clear: on CPU-only hardware, stick to 2-4B parameter models. The jump from 4B to 8B cuts your speed in half while increasing wait times dramatically.
Beyond the CLI: Testing GUI Frontends
While the raw performance benchmarks above were collected via the Ollama CLI (using --verbose for timing), we also ran the same models through a couple of popular GUI frontends that use Ollama as the backend:
- Chatbox
- Open WebUI
These tools don’t change the underlying model weights, but they do add their own system prompts, templates, and chat wrappers. That extra layer can influence how a model presents itself — which is exactly what surfaced in the identity tests below.
The Identity Problem
During testing, we discovered something unexpected: smaller models sometimes forget who they are.
| Model | CLI Response | Open WebUI | Chatbox |
|---|---|---|---|
| gemma2:2b | “Gemma” | “Gemma” | “Gemma” |
| phi3:mini | Correct | “Turing” | “Miles” |
| llama3.2 | “Llama” | “OpenAI” | Generic |
| llama3 (8B) | “LLaMA” | “LLaMA” | Generic |
| gemma4:e4b | “Gemma 4” | “Gemma 4” | “Gemma 4” |
Phi3 calling itself “Miles” or “Turing”? Llama claiming to be “OpenAI”? This happens because:
System prompts from interfaces can override model identity
Smaller models are more susceptible to prompt influence
The same model behaves differently across interfaces
This isn’t a critical problem — the model still works — but it’s a reminder that these are statistical text generators, not entities with fixed identities. Gemma models (both 2:2b and 4:e4b) showed the most stable identity across all interfaces.
Understanding the Numbers
What Does tok/s Actually Mean?
Tokens are the units LLMs process — roughly 0.75 words in English, slightly less in other languages. So:
15 tok/s ≈ 11 words per second — feels like fast typing
8 tok/s ≈ 6 words per second — noticeable but acceptable
5 tok/s ≈ 4 words per second — painfully slow for interactive use
For reference, ChatGPT streams at roughly 30-50 tok/s. Our best local result (gemma2:2b at 15 tok/s) is about half that speed — which is perfectly usable.
Why Is Gemma 4 So Slow Despite Similar tok/s?
Gemma 4’s generation speed (7.98 tok/s) isn’t dramatically lower than gemma2:2b (14.84 tok/s). But the total time is 18x longer (37s vs 2s). Why?
The answer is token count. Gemma 4 generated 278 tokens (including its visible thinking process) versus 15 tokens from gemma2:2b. More tokens × slower speed = much longer wait.
This is the tradeoff with “thinking” models: you get transparency and often better reasoning, but at the cost of time.
The ~15 tok/s Ceiling
On this hardware configuration (i7-1165G7, 20 GB RAM, no GPU), 15 tokens per second appears to be the practical maximum for fluent generation. You can push slightly higher with smaller models, but not by much.
This ceiling is determined by: – CPU compute speed: The i7-1165G7 has 4 cores running at 2.8 GHz – Memory bandwidth: DDR4-3200 provides ~50 GB/s – Model architecture: Transformer attention is computationally expensive
To exceed this ceiling, you need either: – A faster CPU (more cores, higher clock) – A dedicated GPU with VRAM – A different runtime (we’ll explore this in Part 4)
How much difference does the GPU actually make? I measured it directly when I replaced an RTX 3060 with an RTX 3090: generation speed on long prompts more than doubled, from 33–40 to 76–86 tok/s, and Flux image generation dropped from 92 to 44 seconds per frame — most of the gain coming from the extra VRAM, which lets models run entirely in GPU memory instead of offloading part of the work to the CPU.
Practical Recommendations
For Daily Use: gemma2:2b
If you want one model that just works, this is it. Fast enough for interactive chat, capable enough for most tasks, small enough to load quickly. Install it and forget about the others.
ollama pull gemma2:2b
For Complex Reasoning: gemma4:e4b
When you need the model to think carefully — complex analysis, multi-step problems, tasks where you want to verify the reasoning — gemma4:e4b is worth the wait. The visible chain-of-thought is both educational and useful for catching errors.
ollama pull gemma4:e4b
For Experimentation: phi3:mini or llama3.2
Different models have different strengths. Phi3 tends toward technical detail; Llama toward conversational flow. If gemma2:2b doesn’t click with you, try these alternatives.
ollama pull phi3:mini ollama pull llama3.2
Skip Unless Necessary: llama3 (8B)
The 8B model is more capable, but 5.54 tok/s is below the threshold for comfortable interactive use. Consider it for batch processing or offline tasks where you can queue prompts and come back later.
What’s Next
These benchmarks measure raw speed on a simple prompt. But how do these models perform on real tasks?
In Part 3, we’ll stress-test gemma2:2b and gemma4:e4b with practical challenges: – Coding: Write a Python function with edge case handling – Reasoning: Solve a multi-step logic problem – Summarization: Extract key points from a medical discharge letter – Clinical decision support: Generate differential diagnoses from symptoms
We’ll use Chatbox as our interface and discover where small local models succeed — and where they fail.
In Part 4, the plot twist: what if I told you there’s a way to run gemma4:e4b in 5 seconds instead of 37? Stay tuned.
Summary
gemma2:2b is the best choice for CPU-only hardware: 15 tok/s, 2-second responses
~15 tok/s is the ceiling on this hardware configuration
2-4B models are the sweet spot; 8B models are too slow for interactive use
gemma4:e4b offers visible reasoning but at significant time cost
Model identity is unstable on smaller models across different interfaces
Speed ≠ capability: slower models may produce better answers, just… slowly
The numbers are clear. In the next article, we’ll see if they hold up under real-world pressure.
Hardware reference: Intel Core i7-1165G7, 20 GB DDR4 RAM, no dedicated GPU. All benchmarks conducted via Ollama CLI with
--verboseflag. Results may vary on different configurations.
Next: Local AI: Stress-Testing Local Models on Real Tasks — Coding, reasoning, and medical applications
Previous: Local AI: Running LLMs Without Dedicated Graphics— Setup and installation
See also: Local Artificial Intelligence in Healthcare
About this series: “Local AI” documents a practical exploration of running LLMs on CPU-only consumer hardware. All tests were conducted on a 2021-era Intel laptop with 20 GB RAM and no dedicated graphics card. The goal is demonstrating feasibility, not competing with high-end setups.