Local AI: Stress-Testing Local Models on Real Tasks

Part 3 of the “Local AI” series — Coding, reasoning, medical summarization, and clinical decision support

Benchmarks tell you how fast a model runs. They don’t tell you if it can actually do anything useful.

In Part 1 we set up Ollama, and in Part 2 we measured generation speeds. Now we push these models with real tasks: write working code, solve a logic problem, summarize a medical document, and generate differential diagnoses from clinical findings.

The results are revealing. A 2-billion parameter model running on a laptop CPU can produce surprisingly competent output — but it also fails in predictable ways. Understanding these failure modes is essential before relying on local LLMs for serious work.

Test Setup

Models tested:

• gemma2:2b (1.7 GB) — our speed champion from Part 2
• gemma4:e4b (3.0 GB) — the “thinking” model with visible reasoning

Interface: Chatbox Desktop App

We chose Chatbox over Open WebUI for these tests because it’s lighter weight and displays token counts — useful for understanding model behavior. Chatbox connects to Ollama running in the background and provides a clean chat interface.

Hardware: Intel i7-1165G7, 20 GB RAM, no dedicated GPU (same as previous tests)

Why Chatbox?

Several interfaces can connect to Ollama. Here’s why we picked Chatbox for complex testing:

For our purposes — running tests and seeing exactly what the model outputs — Chatbox’s simplicity is an advantage. If you need document upload (RAG) or multi-user access, Open WebUI is the better choice.

Download: chatboxai.app

After installing, select “Ollama” as the provider. Your downloaded models appear automatically in the model selector.

The Four Tests

We designed four prompts that stress different capabilities:

1. Coding: Write a function with edge case handling
2. Reasoning: Solve a multi-step math problem
3. Summarization: Extract key points from a medical document
4. Clinical reasoning: Generate differential diagnoses

All prompts are in English to avoid translation overhead and ensure reproducibility.

Test 1: Coding

The Prompt

Write a Python function that calculates the average heart rate from
a list of BPM values, excluding outliers (below 40 or above 200).
The function should also return the valid minimum and maximum values.

This tests: basic Python, list comprehension, edge case handling, documentation.

gemma2:2b Result

Time: ~1 minute 19 seconds | Tokens: 565 | Speed: 10.56 tok/s

import statistics

def calculate_average_bpm(bpm_values):
  """Calculates the average heart rate from a list of BPM values, excluding outliers."""

  # Filter out outliers below 40 and above 200.
  valid_values = [bpm for bpm in bpm_values if 40 <= bpm <= 200]

  if len(valid_values) == 0:
    return 0, None, None

  average_bpm = statistics.mean(valid_values)
  valid_min = min(valid_values)
  valid_max = max(valid_values)

  return average_bpm, valid_min, valid_max

# Example usage:
bpm_list = [100, 50, 250, 45, 70, 80, 90, 200]
average_bpm, min_bpm, max_bpm = calculate_average_bpm(bpm_list)
print("Average BPM:", average_bpm)

import statistics

def calculate_average_bpm(bpm_values):
  """Calculates the average heart rate from a list of BPM values, excluding outliers."""

  # Filter out outliers below 40 and above 200.
  valid_values = [bpm for bpm in bpm_values if 40 <= bpm <= 200]

  if len(valid_values) == 0:
    return 0, None, None

  average_bpm = statistics.mean(valid_values)
  valid_min = min(valid_values)
  valid_max = max(valid_values)

  return average_bpm, valid_min, valid_max

# Example usage:
bpm_list = [100, 50, 250, 45, 70, 80, 90, 200]
average_bpm, min_bpm, max_bpm = calculate_average_bpm(bpm_list)
print("Average BPM:", average_bpm)

Evaluation:

• ✅ Correct logic and filtering
• ✅ Handles empty result case
• ✅ Includes docstring and example
• ✅ Uses standard library (statistics.mean)

The 2B model produced working, documented code in under 90 seconds. For a quick coding assistant on constrained hardware, this is genuinely useful.

gemma4:e4b Result

Time: ~5 minutes 30 seconds | Tokens: 1,968 | Speed: 6.49 tok/s

from typing import List, Tuple, Union

def calculate_average_hr(bpm_list: List[int]) -> Tuple[float, Union[int, None], Union[int, None]]:
    """
    Calculates the average heart rate from a list of BPM values,
    excluding outliers (below 40 or above 200).
    """
    MIN_VALID = 40
    MAX_VALID = 200

    filtered_bpm = [bpm for bpm in bpm_list if MIN_VALID <= bpm <= MAX_VALID]

    if not filtered_bpm:
        print("\\n[INFO] Warning: No valid heart rate data found.")
        return 0.0, None, None

    average_hr = sum(filtered_bpm) / len(filtered_bpm)
    valid_min = min(filtered_bpm)
    valid_max = max(filtered_bpm)

    return average_hr, valid_min, valid_max

from typing import List, Tuple, Union

def calculate_average_hr(bpm_list: List[int]) -> Tuple[float, Union[int, None], Union[int, None]]:
    """
    Calculates the average heart rate from a list of BPM values,
    excluding outliers (below 40 or above 200).
    """
    MIN_VALID = 40
    MAX_VALID = 200

    filtered_bpm = [bpm for bpm in bpm_list if MIN_VALID <= bpm <= MAX_VALID]

    if not filtered_bpm:
        print("\\n[INFO] Warning: No valid heart rate data found.")
        return 0.0, None, None

    average_hr = sum(filtered_bpm) / len(filtered_bpm)
    valid_min = min(filtered_bpm)
    valid_max = max(filtered_bpm)

    return average_hr, valid_min, valid_max

Evaluation:

• ✅ Type hints (professional practice)
• ✅ Named constants (MIN_VALID, MAX_VALID)
• ✅ More comprehensive edge case handling
• ✅ Informative warning message
• ⚠️ 3.5x more tokens, 4x slower

The larger model produced more “professional” code — the kind you’d see in a production codebase. But it took over 5 minutes. Whether that tradeoff is worthwhile depends on your needs.

Test 2: Reasoning

The Prompt

A hospital ward has 12 patients. Each room can hold a maximum of 3 patients.
4 patients require isolation (single room). What is the minimum number of
rooms needed to accommodate all patients?

This tests: multi-step reasoning, constraint handling, basic arithmetic.

The correct answer: 4 isolation rooms + 3 shared rooms (for 8 patients at max 3 per room) = 7 rooms.

gemma2:2b Result

Time: ~33 seconds | Tokens: 170 | Speed: 11.27 tok/s

The minimum number of rooms needed is to accommodate all non-isolation patients, since 4 need isolation and the other 8 need a general room.

12 – 4 = 8

You will need 8 rooms to accommodate all patients.

Evaluation:

• ❌ Wrong answer: 8 instead of 7
• ❌ Ignored the constraint that rooms hold 3 patients
• The model subtracted correctly (12-4=8) but then treated each remaining patient as needing their own room

This is a characteristic failure of smaller models: they can execute individual steps but miss constraints that require holding multiple facts in mind simultaneously.

gemma4:e4b Result

Time: ~5 minutes 28 seconds | Tokens: 725 | Speed: 5.86 tok/s

1. Account for Isolation Patients

• Patients: 4
• Requirement: Each needs a single room
• Rooms needed: 4 rooms

2. Account for Remaining Patients

• Remaining: 12 – 4 = 8 patients

3. Rooms for Remaining Patients

• Room capacity: 3 patients/room
• Rooms needed: Ceiling(8 ÷ 3) = Ceiling(2.66) = 3 rooms
• (These 3 rooms hold 3 + 3 + 2 = 8 patients)

4. Total Rooms

• 4 + 3 = 7 rooms

Evaluation:

• ✅ Correct answer: 7 rooms
• ✅ Explicit ceiling function reasoning
• ✅ Verified the math (3+3+2=8)
• ✅ Clear step-by-step breakdown

The larger model succeeded where the smaller one failed. It held all constraints in context and applied the ceiling function correctly. But it took 10x longer.

Test 3: Medical Summarization

The Prompt

Summarize this discharge letter in 5 bullet points highlighting:
diagnosis, procedure performed, complications, discharge medications
changes, and follow-up instructions.

[Full discharge letter for a CABG patient - ~800 words]

We used a real (anonymized) discharge summary for a patient who underwent coronary artery bypass grafting. This tests: long context handling, medical terminology, information extraction.

gemma2:2b Result

Time: ~2 minutes 4 seconds | Tokens: 357 | Speed: 8.6 tok/s

Diagnosis: Coronary artery disease, Hyperlipidemia, Hypertension, Depression, Benign Prostatic hypertrophy

Procedure: Coronary Artery Bypass Graft (CABG) with 3 grafts

Medications: Aspirin, Paroxetine, Simvastatin, Metoprolol, Clonidine…

Discharge Instructions: No driving for one month; no lifting >10 pounds for 10 weeks; monitor for fever >100.5, redness, drainage

Follow-Up: Follow up with doctor for further assessment

Evaluation:

• ✅ Correct diagnoses and procedure
• ✅ Medications listed accurately
• ✅ Key restrictions captured
• ❌ Missing complications: The letter mentioned a small apical pneumothorax and blood transfusion (2 units PRBC) — both clinically significant
• ❌ Missing medication changes: Atenolol was switched to Metoprolol; Terazosin dose was changed

The model extracted the obvious information but missed details that a clinician would consider important. For a quick overview, acceptable. For clinical use, insufficient.

gemma4:e4b Result

Time: ~8 minutes 12 seconds | Tokens: 662 | Speed: 5.88 tok/s

Please note: This summary is for comprehension purposes only and is not a substitute for speaking with the discharge medical team.

• Diagnosis: Coronary Artery Disease (CAD), Hyperlipidemia, Hypertension, Depression, Mild Benign Prostatic Hypertrophy
• Procedure: CABG x 3 (LIMA to LAD, SVG to diagonal, SVG to PDA) for surgical revascularization
• Complications: Small apical pneumothorax after chest tube removal; required 2 units PRBC due to low hemoglobin
• Medication Changes: Terazosin changed from 10mg daily to 5mg at bedtime; statin switched from Zocor to Simvastatin
• Follow-up: No driving for 1 month; no lifting >10 lbs for 10 weeks; watch for fever >100.5, redness, drainage

Evaluation:

• ✅ Detailed procedure (named each graft)
• ✅ Captured both complications (pneumothorax + transfusion)
• ✅ Identified specific medication changes
• ✅ Included appropriate disclaimer
• ✅ Clinically complete summary

The difference is stark. The larger model produced a summary that captures clinically relevant details the smaller model missed entirely. The 8-minute wait may be justified for medical applications.

Test 4: Differential Diagnosis

The Prompt

A 58-year-old male presents with retrosternal chest pain for 2 hours,
radiating to the left arm, diaphoresis, and nausea. BP 150/95, HR 98 bpm,
SpO2 96%. ECG shows ST elevation in V1-V4. Troponin pending.

List the differential diagnoses in order of probability and the
immediate diagnostic workup.

This is a classic STEMI presentation. We’re testing: clinical reasoning, differential diagnosis generation, knowledge of emergency protocols.

gemma2:2b Result

Time: ~4 minutes 33 seconds | Tokens: 807 | Speed: 7.72 tok/s

Differential Diagnoses:

1. Acute Myocardial Infarction (STEMI) — Classic presentation with ST elevation
2. Unstable Angina — If troponin negative
3. Pericarditis — Can cause ST changes, but usually diffuse
4. Pulmonary Embolism — Less likely given ECG pattern
5. Aortic Aneurysm — Can cause radiating chest pain

Workup:

• Troponin levels
• ECG analysis
• Echocardiogram
• Chest X-ray
• Possible angiogram

Evaluation:

• ✅ STEMI correctly identified as top diagnosis
• ✅ Reasonable differential list
• ✅ Appropriate workup mentioned
• ❌ Missing aortic dissection — critical DDx for radiating chest pain
• ❌ No mention of aspirin, anticoagulation, or cath lab activation
• ❌ No urgency conveyed (“time is muscle”)

The model demonstrates medical knowledge but misses critical elements. A physician reviewing this would note the absence of aortic dissection (a can’t-miss diagnosis) and the lack of treatment urgency.

gemma4:e4b Result

Time: ~16 minutes 1 second | Tokens: 1,330 | Speed: 5.78 tok/s

This presentation immediately requires management protocols for suspected Acute Coronary Syndrome (ACS), specifically a STEMI.

Differential Diagnoses:

1. STEMI (Most Probable) — Plaque rupture leading to complete coronary occlusion. V1-V4 suggests LAD territory.
2. Pericarditis/Myocarditis — Can cause ST elevation, but typically diffuse pattern, not localized.
3. Aortic Dissection — Must always be considered. Look for “tearing” pain radiating to back.
4. Pulmonary Embolism — Consider if pleuritic pain or hypoxia, but doesn’t typically cause this ECG pattern.

Immediate Management (“Time is Muscle”):

• ABCs: Ensure airway, breathing, circulation
• IV Access: Large-bore IVs
• MONA-B: Oxygen (if SpO2 <90%), Morphine, Nitroglycerin, Aspirin (chewed)
• Draw bloods: Troponin, CBC, coags

This patient requires immediate cardiac workup while en route to the cath lab.

Evaluation:

• ✅ STEMI identified with anatomical localization (LAD territory from V1-V4)
• ✅ Aortic dissection included as must-consider diagnosis
• ✅ MONA-B protocol explicitly stated
• ✅ “Time is muscle” urgency conveyed
• ✅ Cath lab activation mentioned
• ✅ Pathophysiological mechanisms explained

This response demonstrates genuine clinical reasoning. The model understands not just what to do but why, and conveys appropriate urgency. The 16-minute response time is impractical for real clinical use, but the quality is impressive for a local model.

Summary: Model Comparison

Key Findings

1. The Quality Gap Is Real

gemma4:e4b consistently outperformed gemma2:2b on complex tasks. The larger model:

• Solved the reasoning problem the smaller model failed
• Captured clinically critical details (complications, aortic dissection)
• Produced more professional, thorough output

2. The Speed Penalty Is Severe

On Ollama with this hardware, gemma4:e4b takes 3-10x longer than gemma2:2b:

• Simple coding: 5.5 min vs 1.3 min
• Clinical reasoning: 16 min vs 4.5 min

Sixteen minutes for a differential diagnosis makes the model impractical for interactive use.

3. Small Models Have Predictable Failures

gemma2:2b struggles with:

• Multi-step reasoning requiring constraint tracking
• Extracting non-obvious details from long documents
• Comprehensive clinical reasoning

It succeeds at:

• Straightforward coding tasks
• Simple information extraction
• Quick queries where depth isn’t critical

4. Neither Model Is Ready for Unsupervised Clinical Use

Even gemma4:e4b’s excellent differential diagnosis response should not be used without physician review. These models are assistants, not replacements. They can miss critical information, hallucinate details, and lack the judgment that comes from clinical experience.

Practical Recommendations

Use gemma2:2b when:

• Speed matters more than depth
• Tasks are straightforward (simple coding, basic Q&A)
• You’ll verify the output anyway

Use gemma4:e4b when:

• Accuracy is critical
• Tasks require multi-step reasoning
• You can wait 5-15 minutes for a response
• Medical or technical precision matters

Or…

What if you could get gemma4:e4b quality at gemma2:2b speed?

In the next article, we reveal something unexpected: running these same tests with different software produced the same quality in 5 seconds instead of 16 minutes. Same model, same hardware, 40-200x faster.

Stay tuned.

Next: Local AI: The LM Studio Surprise — Same model, 200x faster

Previous: Local AI: Choosing the Right Model for Your Hardware — Benchmarks and model selection

Test conditions: Intel i7-1165G7, 20 GB RAM, no GPU. Interface: Chatbox connected to Ollama. All prompts in English. Medical content reviewed by the author (physician).