Does a model lose quality through quantisation?

How much depends on the task. Between 8 and 4 bit, our measurement showed only a few percentage points, and on one task the smaller file was even better. From 2 bit, arithmetic ability fell to a sixth, while German subject knowledge lost 14 percent. The model still sounds competent and still gets the calculation wrong.

What does Q4_K_M mean?

A GGUF quantisation level for llama.cpp: around 4 bits per parameter, with the K methods storing important layers more precisely than unimportant ones. The letters after it (S, M, L) denote variants of that mix. Which mix suits a given task cannot be read off the file name.

Is AWQ better than GGUF?

At the same bit count, both came out level in our measurement: AWQ under vLLM reached 65 percent on the arithmetic test, Q4_K_M under llama.cpp 66. The choice follows the engine, not the quality: GGUF belongs to llama.cpp and Ollama, AWQ to vLLM.

How do I check whether a quantisation is good enough for my task?

With two to three standard benchmarks matching your task, run with the lm-evaluation-harness against your own server. At 200 questions per test, that takes about an hour per level. What matters is a test that checks arithmetic or multi-step work, because that is exactly what breaks first.

LLM quantisation: how it works and how to use it well

Q: Which quantisation should I choose for a local LLM?

4-bit is the standard to start with: Q4_K_M for llama.cpp, AWQ for vLLM. In our measurement, the step from 8 to 4 bit cost no more than four percentage points on any of the three tasks, while saving 41 percent of the memory. Below 3 bit, arithmetic ability collapsed.

Quantisation is the lever that fits a large model onto affordable cards and makes it answer faster. How it works, how to quantise a model yourself, and how to find the level at which quality still holds.

How do I use quantisation correctly?

Quantisation trades precision for memory: the same model in 4 instead of 16 bit needs a third of the space, answers faster and carries more concurrent users. In our measurement, the step from 8 to 4 bit cost no more than four percentage points on any of three tasks. The art lies in finding the level at which your own task still holds, because below 3 bit, arithmetic collapsed.

In brief

Quantisation is the single most effective lever in the whole operation: a third of the memory, double the speed, more users on the same card.
Start at 4 bit (Q4_K_M for llama.cpp, AWQ for vLLM). The step from 8 to 4 bit cost us almost nothing, and the memory saved turns into KV cache and so into concurrent users.
Do not trust the file name. Our 5-bit file did arithmetic worse than the 4-bit file, at two independent providers.
Measure with a task that matches your own work. Coding and arithmetic ranked the top four levels in opposite order.
Never sign off a quantisation with a knowledge test. Knowledge survives down to 2 bit almost intact; arithmetic collapses long before that.
An afternoon is enough for sign-off: around an hour per level for two standard benchmarks against your own server.

Quantisation is the reason local AI works on affordable hardware. In the first part of this series, a model at GPT-5.1 level ran on two consumer cards; in the second, the memory it saved turned into room for concurrent users. Both exist only because of quantisation, and anyone who understands the mechanism gets one more model class out of the same hardware.

Understanding it pays off for a second reason too. On Hugging Face, dozens of ready-quantised files sit for our 27-billion-parameter model, produced by third parties, with names like Q4_K_M or IQ2_XXS. The name states the bits. It does not state what the file can do, and a model quantised too coarsely does not crash, it simply answers wrong more often.

This article therefore covers both: how quantisation works, walked through once ourselves, and what it costs, measured across seven levels of the same model on three tasks.

This third part of the series covers the technique that makes everything before it possible in the first place.

What quantisation is

The weights of a language model are billions of decimal numbers, stored at 16 bits per number. Quantisation stores the same numbers more coarsely, at 8, 4 or even fewer bits. A 27-billion-parameter model thus shrinks in memory from 57 to around 19 GB, and only then fits onto affordable cards at all.

Blunt rounding would destroy the model, because 4 bits can only represent 16 distinct values. The trick is to work in small groups: the weights are split into blocks of 32 numbers, each block stores a finely resolved scale, and each number only records where it falls on that scale. Because neighbouring weights tend to sit in a similar order of magnitude, this works remarkably well, and the loss per number stays small.

How much of that loss actually shows up in the answers is decided elsewhere: in the question of which layers get how many bits. That is best demonstrated directly.

Quantise a model yourself

The whole process takes two commands and needs no GPU. We walked it through for a small model on our own machine:

# 1. Convert the original weights into GGUF format, full precision
docker run --rm -v ~/.cache/huggingface/hub:/hub -v $PWD:/out   ghcr.io/ggml-org/llama.cpp:full   --convert /hub/models--Qwen--Qwen3.5-0.8B/snapshots/<hash>   --outfile /out/f16.gguf --outtype f16

# 2. Quantise to 4 bit
docker run --rm -v $PWD:/out ghcr.io/ggml-org/llama.cpp:full   --quantize /out/f16.gguf /out/q4_k_m.gguf Q4_K_M

The first step took 6 seconds, the second 3.5. 1,475 MB became 506.

More interesting than the result is the log. The tool decides, layer by layer, how coarse each one may become:

token_embd.weight      f16 -> q6_K   485.0 MB -> 198.9 MB
blk.0.attn_qkv.weight  f16 -> q6_K    12.0 MB ->   4.9 MB
blk.0.ffn_gate.weight  f16 -> q4_K     7.0 MB ->   2.0 MB
output_norm.weight     f32  (untouched)

The embedding and the attention get 6 bits, the bulk of the feed-forward layers 4 bits, and the normalisation layers stay at full precision. That is why the log also states 5.50 bits per weight for a file called “Q4”: the name states the budget, and the distribution across the layers is decided by the author of the tool. Keep that sentence in mind, it explains a measurement finding coming up shortly.

The formats: GGUF, AWQ and the others

Quantised files come in several formats, and the choice follows the engine you host with:

Format	Calibration	Where it runs
GGUF Q8_0, Q6_K, Q5_K_M, Q4_K_M	none	llama.cpp, Ollama
GGUF IQ levels (IQ2_XXS …)	mostly with calibration data	llama.cpp, below 3 bit
AWQ	calibration data	vLLM, SGLang
GPTQ	calibration data	vLLM, older standard
bitsandbytes (NF4)	none, quantises on load	transformers, fine-tuning
FP8	none, a real number format	vLLM, often direct from the manufacturer

To read GGUF names: the number states the rough bit budget, the K stands for the mixing technique just shown, and the letter at the end (S, M, L) denotes a variant of it. IQ levels compress below 3 bit and for that rely mostly on an importance matrix built from calibration data.

AWQ and GPTQ go a step further than GGUF: while quantising, they run sample data through the model and specifically protect the weights that matter most for the outputs. bitsandbytes quantises only on load and produces no file at all, convenient for experiments and fine-tuning, secondary for production. FP8, finally, is not a compression technique but a real 8-bit number format that newer cards can compute with directly; manufacturers such as Qwen officially ship some models this way.

Whether the choice of format affects quality is something we measured too, more on that below.

Seven levels, three tasks

We measured Qwen3.6-27B with llama.cpp on one or two RTX 4090s (the largest levels do not fit on a single card), each level with the same three tests: arithmetic problems in text form (GSM8K), coding tasks with an executable solution (MBPP) and German knowledge questions (Global-MMLU). 200 questions each, 100 for German knowledge; the figure given is the share of correct answers. Differences under around 7 points (German: 10) are not reliable at these sample sizes.

Level	File	Arithmetic	Coding	German knowledge
Q8_0	29.0 GB	70 %	64 %	84 %
Q6_K	22.9 GB	68 %	59 %	84 %
Q5_K_M	19.8 GB	45 %	68 %	83 %
Q4_K_M	17.1 GB	66 %	74 %	82 %
Q3_K_M	13.8 GB	47 %	74 %	83 %
Q2_K_XL	12.0 GB	30 %	47 %	80 %
IQ2_XXS	9.6 GB	12 %	33 %	72 %

Three findings sit in this table, and each contradicts a common assumption.

From 8 to 4 bit, almost nothing happens. Four points on arithmetic, two on German, and on coding the smaller file even does better. In return, the file shrinks from 29 to 17 GB, and the memory saved turns into KV cache on the card, meaning concurrent users.

The cliff sits between 3 and 2 bit. Q3_K_M still codes exactly as well as Q4_K_M and already loses a third on arithmetic. From Q2, it all falls: IQ2_XXS still solves 12 percent of the arithmetic tasks, so almost none.

And the model never crashes while this happens. Why the collapse still goes unnoticed by anyone who only skims the answers is the most important finding of the whole measurement series, more on that shortly.

This ladder comes from a single model. Whether the cliff sits in the same place for other models is something we have not measured; where it sits for yours, only your own measurement will tell you.

A larger file can answer worse

The outlier in the table is Q5_K_M: 45 percent on arithmetic, twenty points below the smaller Q4_K_M file. A value like that is normally a measurement error, so we checked: three independent runs of the same file (45, 44, 47 percent), then the same level from a second provider (51 percent). No broken download. At two providers, this level is worse for this model than the level below it.

We cannot prove the cause, but you have already seen the mechanism above in the quantise log: the name states the bit budget, the distribution across the layers is chosen by the tool. Our model is a hybrid with recurrent layers, and the Q5 mix presumably hits exactly those unfavourably.

Coding and arithmetic rank the same files in opposite order

Read the table once looking only at the arithmetic column, and once looking only at the coding column. On arithmetic, the largest file wins (70 percent); on coding, the fourth-smallest does (74 percent), and the second-largest sits there at 59 percent, behind every file down to 3 bit.

Anyone who had only run the arithmetic test would have bought Q8_0 and left ten points on the table on coding compared with the best file. Anyone who had only run the coding test would have taken Q3_K_M and lost a third on arithmetic.

That is why no single benchmark serves as a sign-off. You need two to three, and they must match your own task: anyone generating code measures code. Anyone answering German subject-matter questions measures German subject-matter questions. A foreign benchmark on a foreign task does not answer your own question.

Knowledge survives, arithmetic does not

The German column behaves differently from the other two across the whole ladder, and that is the most important finding of the measurement series:

	Q8_0	IQ2_XXS	Loss
Arithmetic	70 %	12 %	to a sixth
Coding	64 %	33 %	almost halved
German knowledge	84 %	72 %	12 points (14 percent)

From 8 to 2 bit, the model loses almost all of its arithmetic ability and keeps almost all of its knowledge. An over-quantised model still knows what the revFADP is, still writes cleanly, and can no longer chain three reasoning steps together correctly.

For sign-off, that means: a knowledge test clears a model that can no longer think. Anyone checking whether a quantisation is fit for purpose has to make it do arithmetic or multi-step work, because that is where it breaks first.

”Native” is not a promise of quality

Alongside community files, some manufacturers offer their own quantised versions. Qwen ships the 27-billion-parameter model officially in FP8, meaning 8 bits per parameter, produced by the manufacturer itself. That sounds like the safe choice next to a 4-bit file from the community.

Measured, the community won on two axes and drew level on the third:

	AWQ (community, 4-bit)	FP8 (manufacturer, 8-bit)
Memory per card	9.7 GB	14.5 GB
Speed at one user	65.9 tok/s	30.3 tok/s
Arithmetic	65 %	62 %

Twice the bits, half the speed, one and a half times the memory, and the quality difference sits within the measurement error. We did not isolate exactly what drives the speed gap; it is likely the engine’s 4-bit kernels, which run particularly fast on our RTX 4090s. On a different card generation, the same choice can turn out differently.

This comparison also delivers cross-validation for the whole measurement series, incidentally: AWQ under vLLM (65 percent) and Q4_K_M under llama.cpp (66 percent) are two different 4-bit methods on two different engines, and they land at the same value. The measurement itself is stable, which makes the Q5 outlier stand out all the more.

The same caution applies in the opposite direction when buying: hosts often run a quietly quantised version under a model’s name. What that means for price comparisons is covered in the article on LLM costs.

How to measure it yourself

Ready-made benchmarks are enough as long as they match the task. For that there is the lm-evaluation-harness from EleutherAI, which measures against OpenAI-compatible servers, including your own. With two tests of 200 questions each, one level costs around an hour, and the whole sign-off across three levels costs an afternoon.

01

Choose two to three benchmarks that match the task

MBPP or HumanEval for code, GSM8K for arithmetic and multi-step work, Global-MMLU for German subject knowledge. At least one of them must test reasoning, not knowledge.
02

Run with 200 questions per test

That bounds the standard error to around 3.5 points: good enough for a 20-point outlier like ours, too coarse for fine comparisons. For choosing between two levels, it is enough.
03

Measure the candidate levels through, same settings

Q4_K_M as the starting point, plus the level above and below it. Change nothing between runs except the file, otherwise every difference traces back to the settings.
04

Remeasure at every change

New model version, new provider of the quantisation, new engine version: run the same pass again. The Q5 case shows that this cannot be predicted in advance.

Two traps cost us a failed run each along the way, both producing wrong numbers with no error message. Coding benchmarks execute the generated code and need two explicit approvals for that; if one is missing, the test appears to run through and delivers an empty table. And a reasoning model silently spends its answer budget in the thinking part: without reasoning switched off, the result comes out at exactly 0.0, which looks like a broken model and is a misconfigured measurement setup. The commands that avoid both traps are in the appendix.

With quantisation, the model is chosen and tested. The fourth part of the series covers which software delivers it, and the fifth works out what the whole setup costs.