Quantization: Model quantization

Quantization (Quantization) represents model parameters or intermediate data through low-precision numerical values to reduce memory usage, data transmission volume, and improve inference efficiency on suitable hardware.

1. Classification of quantization techniques

Quantization
├── quantization object
│   ├── Weights
│   ├── Activations
│   └── KV Cache
│
├── Numeric format
│   ├── INT8 / INT4
│   ├── FP8 / FP4
│   └── NF4
│
├── quantization granularity
│   ├── Per-tensor
│   ├── Per-channel
│   └── Per-group
│
├── quantization method
│   ├── PTQ
│   └── QAT
│
├── quantization algorithm
│   ├── GPTQ
│   ├── AWQ
│   └── SmoothQuant
│
└── Tools and formats
    ├── bitsandbytes
    ├── GGUF
    └── GPTQ / AWQ Model Files

Need to distinguish:

2. Symmetric and Asymmetric Quantization

Symmetric Quantization

Use a symmetrical range of integer values, with zero corresponding to integer zero.

For example:

Float Range: -1.0 ~ 1.0
INT8 Range: -127 ~ 127

quantization:

$$q=\operatorname{round}(x/s)$$

Inverse quantization:

$$\hat{x}=q\cdot s$$

Among them:

• (x): original value.
• (q): quantized integer.
• (s)：Scale。
• $\hat{x}$: The approximate value recovered.

Asymmetric Quantization

Zero Point is introduced to allow integer intervals to accommodate asymmetric numerical distributions.

$$q=\operatorname{round}(x/s)+z$$ $$\hat{x}=(q-z)s$$

where (z) is Zero Point.

Actual quantization also requires performing clipping based on the target integer range.

There is no absolute advantage or disadvantage between the two methods, and it depends on the numerical distribution, hardware support and specific quantization algorithm.

3. Quantization Granularity: quantization granularity

Per-tensor

The entire Tensor uses a set of quantization parameters.

The problem is that the range of values ​​for different Channel can vary greatly.

Channel A: -0.1 ~ 0.1
Channel B: -10  ~ 10

If Scale is shared, Channel A may lose a lot of accuracy.

Per-channel

Each Channel uses independent quantization parameters.

Channel 1 → Scale 1
Channel 2 → Scale 2
Channel 3 → Scale 3

Which dimension of the weight matrix actually corresponds depends on the quantization implementation.

Per-group / Group-wise

Divide the weights into subgroups, using independent Scale for each group.

128 Weights → Scale 1
128 Weights → Scale 2
128 Weights → Scale 3

Smaller Group Size is generally easier to adapt to local numerical distributions, but requires more metadata.

Common Group Size include 32, 64, 128, etc.

4. Weight-only and Weight-Activation Quantization

W4A16

Weights     → INT4 / 4-bit Format
Activations → FP16 / BF16

Mainly used to reduce:

• Model weight occupation.
• Video memory bandwidth requirements.
• The amount of data loaded and transferred by the model.

Many 4-bit LLM inference schemes fall into this category.

W8A8

Weights     → INT8
Activations → INT8

Not only compressing weights, but also trying to use low-precision matrix operations to improve computational efficiency.

Why is Activation harder to quantify?

Weight is usually fixed after training, and the numerical distribution can be analyzed in advance.

Activation changes with the input content, Token and model layer, and dynamic Outlier may occur.

Activation Quantization therefore often requires more careful calibration or dynamic quantization strategies.

5. PTQ, QAT and Calibration

PTQ：Post-Training Quantization

Quantize after the model is trained.

Common methods:

• GPTQ。
• AWQ。
• SmoothQuant。

There is no need to retrain the full model.

QAT：Quantization-Aware Training

Simulate quantization errors during training to adapt the model to low-precision calculations.

High-precision training states suitable for gradient updates are usually still retained.

Calibration Dataset

Some PTQ methods require representative samples for analysis:

• Activation distribution.
• Outlier。
• Weight sensitivity.
• quantization error.

For example, provide hundreds to thousands of representative texts for calibration.

The closer Calibration Dataset is to the actual usage scenario, the better it is usually to obtain reasonable quantized results.

But not all quantization methods require Calibration Dataset.

6. GPTQ

GPTQ is a typical post-training weight quantization algorithm.

Core idea:

When quantizing weights, approximate second-order information is used to measure the quantization error and compensate for weights that have not yet been quantized.

Original Weights
W1 W2 W3 W4
     ↓
Quantize W1
     ↓
Estimate Error
     ↓
Adjust Remaining Weights
     ↓
Continue Quantization

Main features:

• Commonly used for 4-bit Weight-only Quantization.
• Calibration data is often required.
• The quantization process can be time-consuming.
• Better model quality can be maintained at lower bit widths.

The advantage of GPTQ lies in quantization error control rather than guaranteeing optimal inference speed on any hardware.

7. AWQ

AWQ：Activation-aware Weight Quantization。

Core idea:

Different weights in the model have different importance for the final output. It is not enough to just observe Weight Magnitude. The characteristics corresponding to Activation must also be considered.

AWQ identifies sensitive weights or Channel via Activation Statistics and reduces their quantization error using methods such as Scaling.

Calibration Data
       ↓
Activation Statistics
       ↓
Identify Salient Channels
       ↓
Protect Sensitive Weights
       ↓
Quantized Model

Compare with GPTQ:

GPTQ and AWQ are both common low-bit inference choices, and actual performance depends on the model, hardware, and inference engine.

8. SmoothQuant

SmoothQuant is mainly used to solve problems that Activation is difficult to quantify.

Observations:

Weights     → Relatively easy to quantify
Activations → Outlier more obvious

The core idea is to transfer part of the numerical range pressure of Activation to the Weight side through equivalent transformation.

Original:
Activation Range → Large
Weight Range     → Moderate

After Smoothing:
Activation Range → Smaller
Weight Range     → Larger

Thus making W8A8 Quantization easier to implement.

Compared with the typical Weight-only scheme of GPTQ and AWQ, SmoothQuant focuses more on the low-precision calculation of Weight + Activation.

9. bitsandbytes

bitsandbytes is a low-precision calculation and quantization tool library, often used in conjunction with Hugging Face Transformers and PEFT.

Typical uses:

• 8-bit model loading.
• 4-bit model loading.
• NF4 / FP4。
• QLoRA fine-tuning.

4-bit NF4 loading

import torch
from transformers import (
    AutoModelForCausalLM,
    BitsAndBytesConfig,
)

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "your-model-id",
    device_map="auto",
    quantization_config=quant_config,
)

Here:

• load_in_4bit: Enable 4-bit weight quantization.
• bnb_4bit_quant_type: Select NF4.
• bnb_4bit_compute_dtype: Set the relevant calculation precision.
• bnb_4bit_use_double_quant: Quantized partial quantization metadata.

Actual GPU support depends on bitsandbytes, PyTorch and the device environment.

bitsandbytes is mainly suitable for low-precision loading of the Hugging Face ecosystem and QLoRA, but it does not mean that it is necessarily the best solution for high-concurrency production inference.

10. NF4, FP4 and Double Quantization

NF4：NormalFloat 4-bit

NF4 is a non-uniform 4-bit representation designed for approximately normally distributed weights.

Normal uniform quantization usually has a fixed value interval, while NF4 uses quantization levels that fit a specific distribution.

It is especially important in QLoRA.

FP4

FP4 is a general term for a class of 4-bit floating point representations.

Note that the encoding of FP4 may differ between implementations. FP4 in bitsandbytes and the other FP4 formats supported by modern GPU are not directly considered the same format.

Double / Nested Quantization

After weight quantization, parameters such as Scale also need to be stored.

Double Quantization further compresses some quantization metadata.

Original Weights
      ↓
4-bit Quantization
      ↓
Weights + Scales
      ↓
Quantize Scales
      ↓
Reduce Metadata Overhead

It reduces the cost of quantified metadata, rather than re-quantifying all the weights that have already been quantified.

11. GGUF Quantization

GGUF is a commonly used model file format in the llama.cpp ecosystem.

Common quantization types:

Q4_K_M and Q5_K_M

K-Quant uses block quantization, grouping Scale, etc. to store weights.

Among them:

• Q4 / Q5 represents the main quantization bit width.
• K means K-Quant series.
• _S / _M represent different model-level quantized combination strategies.

Q4_K_M and Q5_K_M may use different quantization precisions for different Tensors, and not all Tensors are strictly the same bitwidth.

So the actual model size is larger than simple:

Parameters × 4 bits

How to choose the quantified version?

For the same model, it can generally be understood like this:

Q4_K_M → Pay more attention to the balance of memory and quality
Q5_K_M → Willing to increase memory in exchange for higher accuracy
Q6_K   → Further reduce quantization error
Q8_0   → Usually closer to the original high-accuracy model

This is not an absolute quality ranking across models. Practical results require comparing different quantized versions of the same model.

quantized reference data provided by llama.cpp shows that Q5_K_M generally has larger file sizes and lower Perplexity losses than Q4_K_M.

12. FP8 and FP4

FP8

FP8 common format:

E4M3 → 1 Sign + 4 Exponent + 3 Mantissa
E5M2 → 1 Sign + 5 Exponent + 2 Mantissa

E4M3 has higher significant digit precision, E5M2 has wider exponent range.

The floating-point representation is able to cover a wider dynamic range than INT8, but still requires reasonable Scale management.

Modern data centers GPU Hardware support for FP8 continues to improve, so it is increasingly important in large-scale inference and training.

FP4

FP4 can further reduce the weight and data volume of Activation.

However, the actual performance of FP4 is highly dependent on whether GPU has the corresponding low-precision computing hardware, quantization strategy, and Kernel.

Just because FP4 uses 4 bit, it doesn't mean it must be twice as fast as FP8.

13. Dequantization and inference speed

Dequantization restores the quantized value to an approximate floating point value, or completes the equivalent data conversion and calculation inside the calculation Kernel.

For example:

INT4 Weights
     ↓
Read Compressed Weights
     ↓
Dequantization / Fused Kernel
     ↓
Matrix Computation

Why might W4A16 be faster?

Assume that the model originally had 16 GB weights of BF16:

BF16 Weights ≈ 16 GB
INT4 Weights ≈ 4 GB + Metadata

For Decode of Memory-bound, reducing the number of weighted reads may significantly improve speed.

Why is quantization not necessarily faster?

Actual speed depends on:

• GPU supports corresponding low-precision operations.
• Is there a well-optimized Quantized Kernel.
• Dequantization overhead.
• Batch Size。
• Prefill / Decode ratio.
• Kernel Fusion。
• Memory Bandwidth。

If a GPU has a strong BF16 Tensor Core implementation and the chosen INT4 Kernel implementation is poor, then INT4 may only save video memory but not improve speed.

Therefore, memory savings, model quality and actual inference performance need to be measured separately.

14. KV Cache Quantization

Quantization can be used not only for Weights but also for KV Cache.

For example:

Model Weights: INT4
KV Cache:      FP8 / INT8
Activations:   BF16

Main values of KV Cache Quantization:

• Reduce the memory footprint of long contexts.
• Increase the number of concurrencies that can be supported.
• Reduce the amount of data Attention reads from KV Cache.

The cost is that additional precision errors and quantization calculation overhead may be introduced.

For long context and high concurrency inference, the meaning of KV Cache Quantization may be very obvious.

15. quantization scheme and inference engine

Whether the model can run is determined not only by the quantization method, but also by the model file format and Kernel supported by the inference engine.

Specific support will change depending on the inference engine version, model architecture, and hardware.

cannot simply treat GGUF, GPTQ and AWQ files as interchangeable model files.

16. How to evaluate quantized effects

quantization requires comparing three dimensions simultaneously.

It is recommended to compare under the same conditions:

Same Model Architecture
Same Evaluation Dataset
Same Hardware
Same Input / Output Length
Same Concurrency
       ↓
BF16 vs INT8 vs INT4

Quality

You can observe:

• Perplexity。
• Benchmark Accuracy。
• Code generation accuracy.
• Tool calling accuracy rate.
• Long context task performance.

Quantization error may affect different tasks to varying degrees, so you cannot rely solely on a single Benchmark to judge model quality.

17. LoRA, QLoRA, Distillation and Pruning

These techniques are related to quantization, but solve different problems.

LoRA

LoRA (Low-Rank Adaptation) is an efficient parameter fine-tuning method.

Assume that a certain weight matrix is (W), freeze the original matrix during training, and add low-rank updates:

[ W'=W+BA ]

Among them:

• (A), (B): trainable low-rank matrices.
• Rank (r): Controls the capacity of low-rank updates.
• Original weights (W): usually remain frozen.

Therefore, fewer new parameters need to be trained.

QLoRA

Combine quantization with LoRA:

Base Model
    ↓
4-bit NF4
    ↓
Frozen Base Weights
    ↓
Train LoRA Adapters

The main purpose is not to make model inference faster, but to reduce the memory requirements during fine-tuning of large models.

Distillation

Use Teacher Model to train Student Model.

Common ways:

• Output Distillation。
• Logit Distillation。
• Reasoning / Trace Distillation。

The goal is usually to transfer some capabilities of a large model to a smaller model.

Pruning

Remove or sparsify unimportant parameters, channels, or structures.

Includes:

• Unstructured Pruning: Reset partial weights to zero.
• Structured Pruning: Delete Head, Channel, Neuron or Layer.

Note: sparsification does not equal a certain acceleration, the actual speed depends on the hardware and Sparse Kernel support.

Technology comparison

Summary

The most important knowledge for quantization can be summarized as:

Actual selection can start from the scenario:

Local Inference
    ↓
GGUF + llama.cpp / Ollama

GPU Weight-only Inference
    ↓
GPTQ / AWQ Other compatible solutions

GPU FP8 Inference
    ↓
Hardware-supported FP8 Kernels

Low-memory Fine-tuning
    ↓
NF4 + QLoRA

Core principles:

quantization mainly saves memory and data transmission by reducing numerical precision. However, whether it can be accelerated, how much it is accelerated, and how much quality loss is depends on the specific model, quantization scheme, hardware, and inference engine.

---

Reference documentation

• Hugging Face Quantization
• bitsandbytes Documentation
• llama.cpp Quantization
• QLoRA Paper
• AWQ Paper
• GPTQ Paper
• SmoothQuant Paper