Local LLM Parameters Explained

Table of Contents

Abstract

Running a language model locally gives you a kind of control that hosted chatbots often hide. LM Studio, Ollama, llama.cpp, vLLM, and similar tools expose settings such as temperature, top_p, top_k, min_p, repeat_penalty, and context length. The problem is that their names say very little about what an ordinary user will actually notice on screen.

This guide starts from those visible effects. Why does a model repeat itself? Why does it become imaginative but unreliable? Why does it stop before giving the answer? Why does increasing the context sometimes cause an out-of-memory error? For each important parameter, I explain what it controls in plain language, show a small example, add the technical mechanism for readers who want it, and indicate when the setting should — and should not — be changed.

The central message is simple: there is no universal “best configuration.” A good setup depends on the model, the task, the inference engine, and the available hardware. More importantly, many parameters interact. Changing several at once often makes the result harder, not easier, to understand.

The safest workflow: start with the settings recommended by the model developer, run a fixed set of representative prompts, change one parameter at a time, and record both output quality and speed.

1. The 60-second mental model

A local LLM does not normally write a whole sentence in one operation. It generates one token at a time. A token may be a word, part of a word, punctuation, or even a space combined with a word fragment.

After the text:

“The capital of France is”

the model might assign approximate probabilities such as:

Possible next token	Model probability
`Paris`	92%
`located`	3%
`a`	2%
all other tokens	3%

The program must then choose one token. It can always take the most probable one, or it can sample among several plausible candidates. Once the token is chosen, it is added to the text and the entire process starts again for the following token.

Most inference parameters intervene at one of four points:

Parameter family	The practical question it answers	Main examples
Randomness	How adventurous should the choice be?	temperature
Candidate filtering	Which tokens are allowed into the draw?	top-p, top-k, min-p, typical-p, top-nσ
Repetition control	How strongly should previously used words or sequences be discouraged?	repetition, frequency, presence, DRY
Resources and limits	How much text can be read or written, and where is computation performed?	max tokens, context, KV cache, GPU offload, batch size

This distinction matters. If the model stops halfway through an answer, changing temperature is unlikely to help: the likely problem is the output-token limit. If it repeats a paragraph, increasing the context window is unlikely to help: repetition control is the relevant family.

1.1 Four rules before changing anything

Check the model card or generation_config.json. Model-specific recommendations take priority over generic presets.
Change one setting at a time. Otherwise you cannot know which change helped.
Use the same test prompts. A configuration cannot be compared fairly if the questions change.
Do not treat sampling as a cure for a weak model or prompt. Parameters can shape a model’s behaviour; they cannot add knowledge or reasoning ability it does not possess.

1.2 Deterministic and stochastic generation

In greedy decoding, the program always selects the token with the highest probability. This is predictable, but it can make text rigid and can sometimes reinforce repetitive loops.

In stochastic decoding, the next token is sampled from a set of candidates. The most probable candidate still has the best chance of being selected, but it is not guaranteed to win every time. This can improve naturalness and variety, while also increasing the chance of an odd or incorrect continuation.

A common misunderstanding is to equate “low randomness” with “truth.” A low temperature can make an answer more stable, but it does not verify facts. It merely makes the model follow its most probable path more consistently.

1.3 Logits, softmax, and the optional mathematical layer

Before probabilities exist, the model produces numerical scores called logits. A softmax function converts those scores into probabilities:

P(x_t = w \mid \mathbf{x}_{<t}) = \frac{\exp(z_{t,w})} {\sum_{w’ \in \mathcal{V}} \exp(z_{t,w’})}

You do not need this equation to configure a model. Its practical meaning is simply: higher logits become more probable tokens, but the program can reshape or filter that distribution before making a choice.

1.4 The sampler chain: order changes the result

Samplers are applied in sequence, not simultaneously. In current llama.cpp documentation, the default order is:

penalties → DRY → top-nσ → top-k → typical-p → top-p → min-p → XTC → temperature

This means, for example, that llama.cpp normally removes candidates with its truncation samplers before temperature redistributes probability among the survivors. The same numerical settings can behave differently in an engine that uses another order.

For most users the practical advice is straightforward: leave the sampler order at its default. Change it only for a controlled experiment and record the exact llama.cpp build, because defaults and flags can evolve rapidly.

2. Sampling parameters: how the next token is chosen

2.1 Temperature: how strongly the model prefers the obvious choice

In plain language

Temperature changes the contrast between high- and low-probability candidates.

A low temperature makes the favourites even more dominant. Answers tend to be stable, conservative, and similar across runs.
A medium temperature allows more variation while usually preserving coherence.
A high temperature makes alternative continuations more competitive. This can help brainstorming or creative prose, but it can also introduce irrelevant details, contradictions, and malformed code.

Think of a weighted lottery. Temperature does not invent new tickets; it changes how unequal their weights are.

Worked example

Suppose the model is completing:

“At dawn, the sky became…”

and its initial distribution is:

Candidate	Initial probability
`bright`	55%
`pink`	25%
`orange`	15%
`metallic`	5%

With a low temperature, bright may dominate even more strongly. With a higher temperature, pink, orange, and perhaps metallic gain a more realistic chance. The exact transformed percentages depend on the logits, so the table is illustrative rather than a numerical prediction.

For a factual prompt, “metallic” may be an unwanted deviation. For science fiction, it may be the interesting choice. The correct temperature therefore depends on the task.

What temperature does not do

It does not check whether a statement is true.
It does not increase the model’s context window.
It does not directly control response length.
It cannot rescue an incorrect chat template or a badly quantised model.

Technical mechanism

Temperature $T$ divides the logits before softmax:

$$
P_T(w) = \frac{\exp(z_w/T)}{\sum_{w’}\exp(z_{w’}/T)}
$$

As $T$ approaches zero, the distribution concentrates on the highest-scoring token. At $T=1$, the relative distribution is unchanged. Values above 1 flatten it. Some programs implement temperature = 0 as a special deterministic mode rather than literally evaluating the equation at zero.

Sensible starting points

Task	Starting temperature	Why
Data extraction, classification, short factual answers	0.1–0.3	Stable format and wording
Code or mathematical work with a conventional instruct model	0.2–0.4	Limits avoidable variation
General conversation	0.6–0.8	Balance between naturalness and stability
Creative writing	0.8–1.1	More varied phrasing and ideas
Broad brainstorming	1.0–1.3	More unusual candidates; results need filtering

These are starting points, not universal rules. Reasoning models often have vendor-recommended temperatures around 0.6–0.8 even for analytical tasks. Follow the model developer’s recommendation first.

When to change it

Lower it if repeated runs wander, invent details, or produce fragile code.
Raise it slightly if every run is nearly identical, formulaic, or stuck in an obvious continuation.
Do not change it first when output is truncated, slow, or out of memory.

2.2 Top-p: keep enough candidates to cover a probability budget

In plain language

Top-p, also called nucleus sampling, starts with the most probable token and keeps adding less probable candidates until their cumulative probability reaches the chosen threshold.

Using the previous example:

Candidate	Probability	Cumulative probability
`bright`	55%	55%
`pink`	25%	80%
`orange`	15%	95%
`metallic`	5%	100%

With top_p = 0.80, the draw is restricted to bright and pink.
With top_p = 0.95, orange is also admitted.
With top_p = 1.0, nothing is removed by top-p.

After filtering, the remaining probabilities are renormalised to total 100%.

Why it is useful

Top-p adapts to the situation. If the model is very certain, only a few tokens may be needed to reach 0.9. If it is uncertain, dozens or hundreds may be included. Unlike top-k, the size of the candidate set is not fixed.

Technical definition

After sorting tokens from most to least probable, top-p retains the smallest set whose cumulative probability reaches $p$:

k^* = \min \left\{ k : \sum_{i=1}^{k} P(w_{(i)}) \ge p \right\}

Starting points

Task	Starting top-p
Precise or structured work	0.85–0.92
General chat	0.90–0.95
Creative prose	0.95–0.98
Disable top-p filtering	1.0

Common mistake

Lowering both temperature and top-p aggressively can over-constrain the model. It may become terse, repetitive, or unable to use a necessary but moderately probable technical term. Start by tuning one, then decide whether a second filter is actually needed.

2.3 Top-k: allow only a fixed number of candidates

In plain language

Top-k keeps only the k most probable tokens at every step.

top_k = 1: only the leading candidate survives; this is effectively greedy selection.
top_k = 20: at most 20 tokens enter the draw.
top_k = 40: at most 40 tokens enter the draw.
top_k = 0 in llama.cpp: top-k is disabled.

If top-p is a probability budget, top-k is a fixed number of seats.

Strength and weakness

The strength is simplicity: absurd tail candidates cannot enter once they fall outside the first k. The weakness is that the same number is used in very different contexts. Forty candidates may be excessive when the next token is obvious and too restrictive when many continuations are equally reasonable.

That is why top-k is often used as a coarse safety rail alongside a more adaptive sampler. Current llama.cpp documentation lists top_k = 40 as its default. Set it to zero if you deliberately want another sampler to determine the candidate set alone.

Starting points

Goal	Top-k
Disable it	0
Conservative guardrail	20–40
Wider guardrail for varied prose	40–100
Deterministic choice	1

2.4 Min-p: exclude candidates that are too weak relative to the leader

In plain language

Min-p compares every token with the most probable token. If a candidate’s probability is less than a chosen fraction of the leader’s probability, it is removed.

Suppose the leading token has a probability of 60% and min_p = 0.10. The threshold is:

$$
0.10 \times 60\% = 6\%
$$

Any token below 6% is excluded. If the leader had only 20%, the threshold would fall to 2%, allowing a broader set of candidates. This is why min-p adapts naturally to how confident the model is.

Difference from top-p

Top-p asks: “Have the selected candidates accumulated enough total probability?”
Min-p asks: “Is this candidate strong enough compared with today’s leader?”

When one token dominates, top-p may have to admit many tiny tail probabilities to reach its cumulative target. Min-p can reject those tiny candidates because they are weak relative to the leader.

Technical definition

\mathcal{V}^{(p_{\min})} = \left\{ w : P(w) \geq p_{\min}\max_{w’} P(w’) \right\}

Starting points and evidence caveat

Goal	Min-p
Disable it	0.0
Gentle tail filtering	0.03–0.05
Stronger precision bias	0.08–0.15

Current llama.cpp documentation lists 0.05 as the default. However, min-p should not be presented as a proven universal replacement for top-p. A 2025 reanalysis challenged the empirical claims in the original min-p paper. It remains a useful mechanism to test, not a guaranteed upgrade. Some model families also explicitly recommend min_p = 0.

2.5 Typical-p: prefer tokens whose surprise is typical for the current context

In plain language

Typical sampling is based on an unusual idea: good language is not always made from the most predictable word. It often uses words whose amount of “surprise” is appropriate for the sentence.

Imagine three classes of continuation:

so obvious that the prose becomes dull or repetitive;
plausible and informative;
so surprising that the sentence loses coherence.

Typical-p tries to favour the middle group. It ranks tokens by how close their information content is to the distribution’s expected information content, then retains enough of them to reach the chosen probability mass.

Because of that ranking, it can sometimes exclude the single most probable token. This is intentional and distinguishes it from top-p.

When it may help

It can be worth testing on a model that repeatedly falls into predictable phrasing during long prose. It is usually disabled (typical_p = 1.0) because the benefit is model- and task-dependent, and stacking it with several other filters makes the configuration difficult to interpret.

Goal	Typical-p
Disabled	1.0
Gentle experiment for long prose	0.95–0.98
Stronger intervention	0.90–0.95

2.6 Top-nσ: an advanced filter for high-temperature experiments

In plain language

Top-nσ works directly on logits. It keeps tokens whose score lies within a chosen number of standard deviations of the best score. Its main attraction is that the retained set is invariant to ordinary temperature scaling: changing temperature redistributes weights inside the set without moving its boundary.

This is useful mainly for controlled high-temperature experimentation. A beginner does not need it for normal chat, coding, or summarisation.

\mathcal{V}^{(n\sigma)} = \left\{ w : z_w \geq \max(z) – n\sigma_z \right\}

Goal	top-nσ
Disabled	-1.0
Conservative experiment	1.0
Broader high-temperature candidate set	1.5–2.0

If you activate it to study its behaviour, disable the other truncation samplers first. Otherwise you may only be observing whichever earlier filter removed the most tokens.

2.7 Tail Free Sampling: why an old command may fail

Tail Free Sampling, often exposed in old examples as --tfs, attempted to locate the point at which the sorted probability curve flattened into an uninformative tail. It has been removed from current llama.cpp parameter documentation. If an old tutorial produces an “unknown argument” error for this flag, your installation is not necessarily broken; the tutorial is outdated.

3. Repetition controls: four mechanisms that are easy to confuse

These settings do not all solve the same problem. Before changing one, identify what “repetition” means in your output.

Visible problem	Most relevant control
The same word is used too often	frequency penalty
The answer refuses to introduce new subjects	presence penalty
Recently used tokens keep recurring	repetition penalty
A phrase, sentence, or paragraph is copied again	DRY

3.1 Repetition penalty: discourage recently seen tokens

In plain language

Repetition penalty reduces the chance of tokens that have appeared within a recent window. A value of 1.0 means no penalty. Values above 1 make reuse progressively less attractive.

Suppose a technical answer has repeatedly used “mitral valve.” A conventional repetition penalty does not understand that the phrase is medically necessary. It only sees previously used tokens and lowers their scores. If the penalty is too strong, the model may replace the correct term with awkward alternatives such as “the valvular structure” or, worse, an inaccurate expression.

This is why more penalty is not always better, especially for code, scientific writing, and reasoning traces.

Technical mechanism

For a previously seen token with logit $z_w$ and coefficient $\alpha$:

$$
z’_w =
\begin{cases}
z_w/\alpha & z_w > 0\
z_w\alpha & z_w < 0
\end{cases}
$$

The asymmetry ensures that the token becomes less likely whether its original logit is positive or negative.

Use case	Repetition penalty
Disabled	1.0
Mild loop prevention	1.05–1.10
Clearly repetitive chat model	1.10–1.20
Code, JSON, exact terminology	1.0–1.05

Treat values above about 1.2 cautiously. Reasoning-model developers may recommend leaving this parameter at 1.0.

3.2 Frequency penalty: the more a token appears, the greater the cost

Frequency penalty grows with the number of times a token has already been used:

$$
z’_w = z_w – \beta c(w)
$$

If a token has appeared once, it receives one unit of penalty; if it has appeared ten times, it receives ten. This promotes lexical variety, but the effect can accumulate badly in a long technical document. Correct key terms naturally appear many times and may eventually become almost unavailable.

Example: in a marketing paragraph, a gentle frequency penalty may stop the model repeating “innovative” in every sentence. In an article about atrial fibrillation, the same mechanism may push it away from the precise term “atrial fibrillation.”

Task	Frequency penalty
Technical or long-form factual writing	0–0.1
General prose needing more varied vocabulary	0.1–0.4
Strong intervention	0.5–1.0, with careful review

3.3 Presence penalty: a one-time cost after first use

Presence penalty applies a fixed cost once a token has appeared, regardless of whether it appeared once or many times:

$$
z’_w = z_w – \gamma\mathbf{1}[c(w)>0]
$$

Its practical effect is more topical than lexical. By making already visited material slightly less attractive, it encourages the model to introduce new ideas or subtopics.

Example: ask for ten healthcare startup ideas. Without a presence penalty, several answers may be variations on remote monitoring. A modest presence penalty can encourage the list to move toward logistics, rehabilitation, clinical documentation, education, or prevention.

Task	Presence penalty
Default	0.0
Broader brainstorming	0.2–0.5
Stronger topic exploration	0.5–1.0, with review

Do not assume the numerical scale is implemented identically by every engine. Check the interface and documentation you are actually using.

3.4 repeat_last_n: the model’s rear-view mirror

repeat_last_n sets how many recent tokens are inspected by the repetition penalty.

A window of 64 may catch “the model repeats the model repeats the model.”
A window of 512 has a better chance of noticing that a paragraph is reusing wording from several paragraphs earlier.
In llama.cpp, -1 means use the entire context; 0 disables this penalty window.

A larger window is not free: more legitimate earlier vocabulary becomes eligible for penalty. It also cannot detect semantic repetition when the wording changes. A model can restate the same idea with synonyms without triggering token-based repetition control.

Output type	Starting window
Short chat reply	64
Essay or report	128–256
Long narrative	256–1024

3.5 DRY: penalise repeated sequences, not repeated vocabulary

In plain language

DRY stands for Don’t Repeat Yourself. Instead of punishing each reused token, it looks for a sequence that is beginning to reproduce an earlier sequence. The longer the match becomes, the faster the penalty grows.

This is a much more targeted response to loops. The word “model” can appear repeatedly without cost, but reproducing the same sentence or paragraph becomes progressively difficult.

Worked example

Earlier text:

“The lighthouse beam crossed the empty harbour every twelve seconds.”

Newly generated text begins:

“The lighthouse beam crossed the empty…”

DRY detects that the new suffix matches an earlier sequence. It penalises the next token that would continue the copy, encouraging a different continuation. Short common phrases can be exempted through dry_allowed_length.

The penalty is commonly expressed as:

$$
\text{penalty} = \text{multiplier}\times\text{base}^{(\ell-\text{allowed length})}
$$

Parameter	Meaning	Current llama.cpp default
`dry_multiplier`	Overall strength; zero disables DRY	0.0
`dry_base`	How fast the cost rises with match length	1.75
`dry_allowed_length`	Short repeats left unpenalised	2
`dry_penalty_last_n`	How far back DRY searches	64

A reasonable first experiment for repetitive prose is dry_multiplier = 0.8. Test it without a conventional repetition penalty so that you can attribute the result.

Do not use DRY blindly for code, JSON, tables, templates, or other structured output. Repeated syntax is legitimate and often mandatory. DRY may “solve” the repetition by breaking the structure.

3.6 XTC: deliberately avoid the most obvious continuation

XTC, or Exclude Top Choices, is a specialised creative-writing sampler. With a chosen probability, it removes some of the most probable tokens above a threshold while retaining a safer option. The aim is to escape clichés without flattening the entire distribution through extreme temperature.

Example: after “The night was as dark as…”, a conventional sampler may repeatedly select familiar comparisons. XTC occasionally blocks the obvious leader and gives another still-plausible continuation a chance.

Use case	XTC probability	XTC threshold
Normal use	0.0	—
Creative experiment	0.3–0.5	about 0.1
Factual answers, medicine, code, mathematics, structured extraction	0.0	—

For factual work, XTC is conceptually the wrong tool: it intentionally interferes with the model’s most likely continuation. Keep it disabled.

4. Length and memory: prompt, output, and context are different limits

4.1 Tokens are not words

Token counts cannot be converted to words with one universal ratio. English prose often uses fewer tokens per word than highly inflected languages, code, formulas, or unusual terminology. As a rough planning estimate only, 1,000 English words might occupy around 1,300–1,800 tokens; Italian or technical text may differ.

Use the tokenizer for the exact model when a document is near the context limit.

4.2 Max output tokens: how much the model may write

This parameter appears under different names:

Engine or library	Common name
llama.cpp	`-n`, `--n-predict`
Ollama	`num_predict`
Transformers	`max_new_tokens`
OpenAI-compatible APIs / vLLM	`max_tokens`

It is a ceiling, not a requested length. Setting 2,000 does not force a 2,000-token answer; the model may stop earlier when it emits an end-of-sequence token or encounters a stop sequence.

Typical failure: a reasoning model receives only 512 output tokens. It spends most of the budget analysing the problem and is cut off before the final answer. Raising the output limit fixes truncation; changing temperature does not.

Expected output	Starting maximum
Short answer	128–256
Several paragraphs	512–1024
Code or detailed report	1024–4096
Reasoning model with a long internal trace	4096–32768, model-dependent

4.3 Context length: the model’s total working space

Context length is the maximum working space shared by several components:

$$
\text{system instructions} + \text{chat history} + \text{new prompt} + \text{generated output} \leq \text{context window}
$$

If the context is 8,192 tokens and the loaded prompt plus history already occupies 7,500, fewer than 692 tokens remain for the answer, allowing for templates and special tokens.

Increasing context lets the model receive more material, but it does not guarantee that it will use distant information reliably. It also enlarges the KV cache and therefore memory consumption. “The model supports 128k” and “128k is a sensible local default” are very different claims.

Current llama.cpp documentation describes -c 0 as loading the context size from the model. Explicit values remain useful when you want a smaller, predictable memory footprint.

4.4 KV cache: the model’s working notes

During generation, the model stores intermediate keys and values for previous tokens so it does not have to recompute the whole conversation at every step. This storage is the KV cache.

A helpful analogy is a desk covered with working notes. A longer context needs a larger desk. Quantising the cache writes those notes more compactly, saving memory at the possible cost of some precision.

For grouped-query attention, a useful theoretical estimate is:

$$
M_{KV}=2\times L\times n_{kv}\times d_{head}\times C\times b
$$

where $L$ is the number of layers, $n_{kv}$ the number of key/value heads, $d_{head}$ the head dimension, $C$ the context length, and $b$ the bytes per stored value. The factor 2 accounts for keys and values.

Do not blindly use a formula based on the full model dimension for a GQA model: it can substantially overestimate cache size. Real memory usage also includes buffers, alignment, batching, architecture-specific features, and backend overhead, so the formula is an estimate rather than a guarantee.

4.5 KV-cache quantisation

Current llama.cpp supports separate cache types for keys and values, including FP16, Q8_0, and Q4 variants.

Cache type	Approximate trade-off
FP16	Highest memory use; conservative quality baseline
Q8_0	Roughly half the raw storage of FP16; usually a good first optimisation
Q4 variants	Much lower storage; greater risk of quality loss, especially in long-context retrieval

When memory is tight, Q8_0 is usually a more cautious first step than jumping directly to Q4. Benchmark retrieval from the middle and end of long documents; casual short-chat testing may not reveal the degradation.

5. Performance parameters: what changes speed and VRAM use

5.1 GPU offload: where the model’s layers are stored and computed

Local engines can place model layers in GPU VRAM, system RAM, or a mixture of both. GPU-resident layers are generally much faster. If some layers remain on the CPU, data must cross the CPU–GPU boundary repeatedly during generation.

The result is often a performance cliff rather than a smooth linear decline. A smaller or more strongly quantised model that fits fully in VRAM can outperform a larger model with only partial offload.

Current llama.cpp exposes -ngl, --gpu-layers, or --n-gpu-layers; current documentation lists auto as the default and also accepts all. Older guides that describe a universal default of zero layers are outdated.

For a rough weight-only estimate:

$$
M_{weights}\approx\frac{P\times Q}{8}
$$

where $P$ is parameter count and $Q$ the average bits per parameter. Actual VRAM must also accommodate the KV cache, compute buffers, model metadata, and backend overhead.

5.2 Batch size: primarily the speed of reading the prompt

Batch size controls how many prompt tokens can be processed together during prefill — the phase in which the model reads your prompt and chat history.

A larger batch can accelerate ingestion of a long document on a capable GPU.
It consumes more memory.
It usually has far less effect on the speed at which new tokens appear one by one after prefill.

If a five-page prompt takes too long to be ingested, batch size may matter. If a short chat prompt loads instantly but generation is slow, changing batch size is unlikely to be the main solution.

Current llama.cpp distinguishes a logical maximum batch (-b, currently documented as 2048 by default) from a physical micro-batch (-ub, documented as 512 by default). Beginners should normally leave both on automatic/default settings unless profiling shows a specific bottleneck.

5.3 CPU threads

Thread count matters mainly when part or all of inference runs on the CPU. More threads do not always mean more speed because LLM inference is often limited by memory bandwidth. On some systems, matching physical performance cores works better than using every logical thread, especially on hybrid P-core/E-core processors.

Test several values and compare tokens per second. There is no thread count that is optimal for every CPU, memory configuration, backend, and model.

5.4 Seed: repeat the same random experiment

A seed initialises the pseudo-random generator. With the same prompt, model, parameters, software build, and execution conditions, a fixed seed can make stochastic output reproducible in practice.

It is not a universal guarantee of byte-for-byte identity. GPU arithmetic, driver versions, batch scheduling, and continuous batching can change numerical details. For a published benchmark, record at least:

the exact model and quantisation;
inference-engine version or commit;
prompt and chat template;
all sampling parameters;
seed;
hardware and backend;
whether requests were batched or concurrent.

5.5 Stop sequences: where generation must end

A stop sequence tells the engine to halt when a specified pattern appears.

Example: in a dialogue formatted as User: and Assistant:, stopping on User: prevents the model from inventing the next user turn. In data extraction, a blank line might be used as a terminator.

Stop sequences are simple but fragile. If the sequence can legitimately occur inside the answer, generation may end too early. For strict JSON, a grammar or JSON schema is stronger because it constrains which tokens may be emitted instead of merely cutting the text afterward.

5.6 Flash Attention

Flash Attention reorganises attention computation to reduce memory traffic and avoid materialising the entire attention matrix. It often improves speed and memory use, especially with longer contexts, but support depends on the backend, GPU, data type, and model architecture.

Current llama.cpp offers -fa on|off|auto and documents auto as the default. Leave it on auto unless you are benchmarking or diagnosing a compatibility problem; “always force it on” is no longer necessary as a general instruction.

6. Mirostat: a thermostat for how surprising the text should be

Static samplers use fixed settings even though the model’s uncertainty changes from one sentence to the next. Mirostat instead tries to keep the observed surprisal near a target over time.

The thermostat analogy is useful:

tau is the desired room temperature — the target level of surprise;
eta is how aggressively the thermostat reacts to a deviation;
the controller continually adjusts the effective truncation rather than leaving it fixed.

Lower tau produces more predictable text; higher tau permits more surprising text. Tau is measured in nats of surprisal, not directly in perplexity, with:

$$
\text{perplexity}=e^{\tau}
$$

In current llama.cpp, enabling Mirostat causes top-k, top-p, and locally typical sampling to be ignored. Do not build a complex preset around parameters that the engine then bypasses.

Goal	Mode	Tau	Eta
Disabled	0	—	—
Balanced experiment	2	5.0	0.1
More predictable	2	3.0	0.1
More surprising	2	7.0	0.1

Mirostat is not necessary for ordinary local use. It is worth testing when a long generation alternates between dull, repetitive passages and unstable, overly surprising passages despite reasonable static settings.

7. Settings recommended by the model developer come first

Generic rules are useful only when the model developer provides no better information. Look in the model card and in generation_config.json. The same family may recommend different settings for thinking and non-thinking modes.

Model family or mode	Temperature	Top-p	Top-k	Min-p	Important note
DeepSeek-R1-0528	0.6	0.95	—	—	Settings used for sampling-based benchmarks; not necessarily a mandate for every task
Qwen3 thinking	0.6	0.95	20	0	The model card warns against greedy decoding and suggests presence penalty for endless repetition
Qwen3 non-thinking	0.7	0.8	20	0	The same weights use different settings when thinking is disabled
Mistral Small 3.2	0.15	—	—	—	Much lower than generic chat advice
Magistral Small	0.7	0.95	—	—	The model card also specifies a 40,960-token output allowance
Phi-4-reasoning	0.8	0.95	50	—	Requires sampling; the model card advises up to 32,768 output tokens for complex tasks

These values are version-sensitive. Before publication or use, follow the link to the exact repository and check whether the model card has changed. Do not transfer settings from one family member to another merely because their names are similar.

8. Which parameter should I change? A symptom-based guide

What you observe	First check	Reasonable first action
Same sentence or paragraph repeats	DRY availability; temperature near zero	Test DRY around 0.8 for prose, or mild repetition penalty if DRY is unavailable
Necessary technical words are replaced by awkward synonyms	Repetition/frequency penalties	Reduce or disable them
Answers are coherent but bland	Vendor settings, temperature	Raise temperature slightly; for creative prose, consider XTC experimentally
Answers become strange or contradictory	Temperature, XTC, penalty strength	Disable XTC; lower temperature; ensure penalties are not excessive
Model stops mid-answer	Output-token limit and stop sequences	Raise max output tokens or remove an over-broad stop pattern
Long prompt causes out-of-memory error	Context and KV cache	Reduce context or test Q8 KV-cache quantisation
Long prompt loads slowly, but generation is acceptable	Batch/prefill configuration	Profile batch and micro-batch sizes
Generation itself is very slow	GPU offload, model size, quantisation	Aim for full GPU offload or use a smaller/stronger-quantised model
JSON or code becomes malformed	DRY, XTC, repetition penalty	Disable creative repetition controls; use grammar/schema-constrained decoding
Same seed gives different output	Build, GPU, batching, concurrency	Record environment; do not promise bitwise determinism

9. Ready-to-use starting profiles

These are baselines for conventional instruct models, not universal recipes. Use the model developer’s values whenever available. In LM Studio, set the corresponding controls in the model/session interface; not every backend exposes every advanced sampler.

9.1 Precise extraction or classification

temperature:       0.2
top_p:             0.90
top_k:             20–40
min_p:             0.00–0.05
repeat_penalty:    1.0
DRY:               off
XTC:               off

Use a strict prompt and, where possible, schema-constrained output. Low temperature improves stability but does not validate the extracted facts.

9.2 Coding

temperature:       0.2–0.3
top_p:             0.90
top_k:             20–40
repeat_penalty:    1.0
frequency penalty: 0.0
DRY:               off
XTC:               off

Repeated identifiers and syntax are necessary, so avoid aggressive repetition control.

9.3 General conversation

temperature:       0.7
top_p:             0.90–0.95
top_k:             40
min_p:             0.05
repeat_penalty:    1.0–1.1
DRY:               off initially
XTC:               off

9.4 Long-form explanatory or academic prose

temperature:       0.3–0.6
top_p:             0.90–0.95
top_k:             20–40
repeat_penalty:    1.0–1.05
frequency penalty: 0.0
DRY:               test 0.8 only if verbatim loops occur
XTC:               off

Keep terminology stable. If the text repeats concepts using different words, improve the outline and prompt before applying stronger token penalties.

9.5 Creative narrative

temperature:       0.9–1.1
top_p:             0.95
min_p:             0.03–0.05
repeat_penalty:    1.0
DRY multiplier:    0.8
XTC probability:   0.3–0.5 (optional experiment)
XTC threshold:     about 0.1

Change DRY and XTC separately at first. Both alter repetition/predictability, but through different mechanisms.

9.6 Reasoning model

temperature:       use the model card; often 0.6–0.8
top_p/top_k:       use the model card
repeat_penalty:    usually 1.0 unless the developer says otherwise
DRY/XTC:           off
max output tokens: generous enough for reasoning plus final answer

Do not automatically copy a low-temperature coding preset onto a reasoning model.

10. Verified command examples

10.1 Ollama

ollama run does not accept ordinary sampling flags such as --temperature 0.2. Use an interactive command, a Modelfile, or the API.

ollama run llama3.1:8b
>>> /set parameter temperature 0.2
>>> Explain the difference between sensitivity and specificity.

FROM llama3.1:8b
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER num_ctx 8192

ollama create my-precise-model -f ./Modelfile

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain the difference between sensitivity and specificity.",
  "options": {
    "temperature": 0.2,
    "top_p": 0.9,
    "top_k": 40,
    "num_ctx": 8192,
    "num_predict": 512
  },
  "stream": false
}'

10.2 llama.cpp

llama-cli -m models/model.Q4_K_M.gguf \
  --temp 0.2 --top-p 0.9 --top-k 40 --min-p 0.05 \
  -c 8192 -n 512 -ngl auto -fa auto \
  -p "Explain the difference between sensitivity and specificity."

Long-context example with an 8-bit KV cache:

llama-cli -m models/model.Q4_K_M.gguf \
  -c 32768 -ctk q8_0 -ctv q8_0 -ngl auto -fa auto \
  -n 1024 -p "[Long document and question]"

Creative prose with DRY:

llama-cli -m models/model.Q4_K_M.gguf \
  --temp 1.0 --top-p 0.95 --min-p 0.05 \
  --repeat-penalty 1.0 \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 \
  -n 2048 -ngl auto -fa auto \
  -p "Continue the story: The lighthouse had been dark for three hundred years."

10.3 vLLM through an OpenAI-compatible endpoint

vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --max-model-len 8192

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain the difference between sensitivity and specificity."}
    ],
    "temperature": 0.2,
    "top_p": 0.9,
    "max_tokens": 512
  }'

10.4 Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{
    "role": "user",
    "content": "Explain the difference between sensitivity and specificity."
}]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
    return_dict=True,
).to(model.device)

set_seed(42)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.2,
    top_p=0.9,
    top_k=40,
)

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

11. A small benchmarking protocol that is better than intuition alone

Create 10–30 prompts that represent your real workload. Include easy cases, difficult cases, long inputs, strict formats, and at least a few prompts where you know the expected answer.

For every configuration, record:

Domain	Suggested measurement
Correctness	Exact match, rubric score, unit tests, or expert review
Format reliability	Percentage of valid JSON / compilable code / complete fields
Repetition	Repeated n-grams, duplicated sentences, or manual rating
Diversity	Distinct ideas or lexical diversity, only when the task needs it
Speed	Prompt-processing tokens/s and generation tokens/s separately
Memory	Peak VRAM and RAM
Stability	Variation across several seeds

Keep the model, prompt template, input order, and hardware fixed. Change one parameter. A single attractive answer proves very little; stochastic settings should be tested across repeated runs.

12. Quick reference

Parameter	Simple meaning	Increase it when…	Reduce it when…	Typical “off” value in llama.cpp
Temperature	Makes alternative tokens more competitive	output is rigid or bland	output drifts or becomes erratic	special case / near 0 for greedy-like use
Top-p	Expands cumulative probability budget	language is over-constrained	unlikely continuations enter	1.0
Top-k	Expands fixed candidate count	too few alternatives survive	long-tail candidates enter	0
Min-p	Raises threshold relative to best token	tail noise appears	useful alternatives disappear	0.0
Typical-p	Broadens locally typical set as it approaches 1	text is over-constrained	testing stronger typical filtering	1.0
Top-nσ	Widens logit-space candidate band	high-temp set is too narrow	candidate set is too broad	-1.0
Repetition penalty	Discourages recently seen tokens	token loops occur	terminology or identifiers get distorted	1.0
Frequency penalty	Penalises repeated use cumulatively	one word is overused	technical terms disappear	0.0
Presence penalty	Encourages unvisited material	brainstorming stays on one theme	answer jumps topics	0.0
DRY multiplier	Blocks continuation of copied sequences	phrases/paragraphs repeat	structured syntax breaks	0.0
XTC probability	Sometimes removes obvious leaders	creative prose is clichéd	task is factual or structured	0.0
Max output tokens	Caps how much may be generated	answer is cut off	latency or verbosity is excessive	—
Context length	Total working space	prompt/history is truncated	VRAM use is excessive	0 = load from model in current llama.cpp
Batch size	Parallelism during prompt ingestion	long prompts load slowly and memory is available	out-of-memory occurs during prefill	—
GPU layers	Amount placed in VRAM	VRAM is available and inference is slow	model does not fit	current default: auto
Seed	Selects repeatable random sequence	you need comparable runs	you want different samples	-1 = random

Defaults vary by engine and version. Check llama-cli --help, the Ollama Modelfile/API documentation, or the corresponding backend documentation for the installed build.

13. Conclusion

Local LLM configuration becomes much easier once the settings are separated by function.

Temperature changes how strongly the model favours its leading candidates.
Top-p, top-k, min-p, typical-p, and top-nσ decide which candidates are allowed to compete.
Repetition, frequency, presence, and DRY penalties address different forms of repetition.
Max tokens and context control output space and total working space.
KV-cache type, GPU offload, batch size, threads, and Flash Attention primarily affect memory and performance.

The goal is not to activate every sophisticated sampler. It is to use the fewest controls needed to solve an observed problem. Start from the model developer’s recommendation, keep a small benchmark set, change one variable at a time, and record the complete configuration. That method is slower than copying a fashionable preset for the first five minutes and much faster over the life of a project.

References

Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020.
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019). CTRL: A Conditional Transformer Language Model for Controllable Generation.
Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2023). Locally Typical Sampling. Transactions of the Association for Computational Linguistics, 11.
Basu, S., Ramachandran, G. S., Keskar, N. S., & Varshney, L. R. (2021). Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity. ICLR 2021.
Nguyen, M. N., Baker, A., Neo, C., Roush, A., Kirsch, A., & Shwartz-Ziv, R. (2025). Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs. ICLR 2025.
Schaeffer, R., Kazdan, J., & Denisov-Blanch, Y. (2025). Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models.
Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017.
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.
Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.
ggml-org. llama.cpp server and CLI parameter documentation. Retrieved September 3, 2026.
Ollama. Modelfile documentation and API documentation. Retrieved September 3, 2026.
Hugging Face. Transformers generation strategies. Retrieved September 3, 2026.
vLLM. Sampling parameters documentation. Retrieved September 3, 2026.