Quantization: how a large model fits on a laptop

A language model that runs on a laptop or a field device has to fit in that machine's memory, because there is no data center to call. Quantization makes it fit by storing each weight with fewer bits. The llama.cpp quantization documentation describes it as reducing the precision of model weights, for example from 32-bit floats to 4-bit integers, which shrinks the model and can speed up inference. The cost is accuracy, and a claim that a model runs on a given machine does not say how much was lost.

For the Llama-3.1-8B model, that document lists the 16-bit floating-point (F16) file at 14.96 gibibytes (GiB) and the Q4_K_M file at 4.58 GiB, at 4.8944 bits per weight.

Bits per weight is not the number in the name

A type name such as Q4_K suggests 4 bits per weight. The stored cost is higher, because weights are quantized in blocks and each block carries extra values. The Hugging Face Hub documentation describes Q4_K as 4-bit quantization in super-blocks of eight blocks, each block holding 32 weights. A weight is reconstructed from its stored integer, a 6-bit block scale, and a 6-bit block minimum, and the table gives the result as 4.5 bits per weight.

File-level figures differ again. The llama.cpp README lists these for Llama-3.1-8B, in bits per weight and GiB:

• F16: 16.0005 and 14.96
• Q8_0: 8.5008 and 7.95
• Q6_K: 6.5633 and 6.14
• Q4_K_M: 4.8944 and 4.58
• Q2_K: 3.1593 and 2.95

The README's --pure option disables what it calls k-quant mixtures and quantizes all tensors to the same type. That wording indicates that the default recipes behind these names mix tensor types, which is why a whole-file figure need not match a single type's figure. The README's table reports size and speed. It reports no accuracy measurement.

What post-training quantization costs

Post-training quantization (PTQ) converts a model that is already trained, without any fine-tuning. A survey by Gholami and colleagues contrasts it with quantization-aware training, which retrains the model. PTQ avoids the retraining, but the survey notes this often comes at the cost of lower accuracy than quantization-aware training, especially at low precision.

The GPTQ paper applies PTQ to GPT-style models. The introduction notes that the parameters of the 175-billion-parameter GPT-3 model occupy 326 GB in float16, counted in multiples of 1024. The abstract reports quantizing models of that size in about four graphics processing unit (GPU) hours, to 3 or 4 bits per weight, with negligible accuracy degradation.

The accuracy cost depends on model size. The paper's Table 3 reports perplexity on the WikiText2 dataset for the OPT family of models. Perplexity is a language-modeling score on which lower is better. For OPT-175B it was 8.34 at 16 bits and 8.37 with GPTQ at 4 bits. For OPT-125M it was 27.65 at 16 bits and 31.12 with GPTQ at 4 bits. At 3 bits, GPTQ gave 8.68 and 53.85 for the same two models.

The paper's baseline, round-to-nearest (RTN) quantization, did worse. At 4 bits it gave 10.54 for OPT-175B, and at 3 bits it gave 7.3e3, a collapse. These figures describe GPTQ and RTN on OPT models. They are not measurements of the GGUF file types above.

GGUF is the container, not the method

GGUF is a file format, not a quantization method. The specification calls it a binary format for storing models for inference with GGML and executors based on GGML, designed for fast loading and saving. A file holds key-value metadata and tensor data, and each tensor entry records its type.

The Hugging Face documentation contrasts GGUF with tensor-only formats such as safetensors: GGUF encodes both the tensors and a standardized set of metadata. The Hub includes a viewer that shows a GGUF file's metadata and each tensor's name, shape, and precision, so the precision of a file can be read from the file itself.

A benchmark score belongs to one file

The llama.cpp README says quantization may introduce accuracy loss, usually measured in perplexity or Kullback-Leibler divergence, and that a suitable importance-matrix file can minimize it. It also warns that requantizing an already quantized model can severely reduce quality compared with quantizing from 16-bit or 32-bit weights.

Dettmers and Zettlemoyer ran more than 35,000 experiments at 3-bit to 8-bit precision, on models from 19 million to 176 billion parameters, across four model families (BLOOM, OPT, NeoX/Pythia, and GPT-2). Their conclusion is that 4-bit precision is almost universally optimal for total model bits and zero-shot accuracy. The finding applies to those families and to zero-shot evaluation. It describes the trade between size and accuracy and does not guarantee the result for any particular model.

Quantization changes the weights, so a benchmark score belongs to the exact file it was measured on. A score for the 16-bit model does not carry over to the 4-bit file that runs on the delivered hardware. In the GPTQ tables above, the gap between 16-bit and 4-bit perplexity is far smaller for the 175-billion-parameter model than for the 125-million-parameter one, and the llama.cpp table reports no accuracy figure at all.

What to check

• Which quantization type is the delivered file, and how many bits per weight does it use?
• Was the accuracy score measured on that exact file or on the 16-bit original?
• Was the file quantized from 16-bit or 32-bit weights, or requantized from an already quantized model?
• Which accuracy measure was used, such as perplexity or Kullback-Leibler divergence, and on what text?
• Does the file's own metadata, as a GGUF viewer shows it, match the type the vendor states?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from. The long read on model provenance goes further.

Sources

1. 1llama-quantize (llama.cpp quantize tool README) · ggml-org (llama.cpp project)
2. 2GGUF (GGML file format specification) · ggml-org (ggml project)
3. 3GGUF (Hugging Face Hub documentation) · Hugging Face
4. 4A Survey of Quantization Methods for Efficient Neural Network Inference · arXiv (Gholami et al.)
5. 5GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers · arXiv (Frantar et al.; ICLR 2023)
6. 6The case for 4-bit precision: k-bit Inference Scaling Laws · arXiv (Dettmers and Zettlemoyer)