GGUF vs GPTQ vs AWQ vs EXL2: LLM Formats Explained
A comprehensive breakdown of popular large language model storage and quantization formats in 2026, including GGUF, GPTQ, AWQ, and EXL2.
Stock photo for illustration only, not from the actual event
- GGUF is a single-file binary format carrying weights, tokenizers, and chat templates for llama.cpp.
- GPTQ and AWQ handle post-training quantization, with AWQ protecting 1% of salient weights.
- EXL2 serves as the native format for ExLlamaV2 optimized for consumer-grade GPUs.
- Safetensors replaced older pickle-based .bin formats to eliminate arbitrary code execution risks.
Understanding how large language models (LLMs) store tensors on disk starts with distinguishing between the container format and the quantization method. Weight memory roughly equals the number of parameters multiplied by bits-per-weight divided by eight, representing pure arithmetic without factoring in KV cache overhead or runtime requirements.
Unquantized models typically ship as 16-bit weights using either pytorch_model.bin or model.safetensors files. Older .bin formats rely on Python pickle, which introduces severe security risks by executing arbitrary code when loading untrusted checkpoints.
Stock photo for illustration only, not from the actual event
To mitigate this vulnerability, Hugging Face introduced Safetensors, consisting of a lightweight JSON header paired with raw tensor buffers containing zero executable code. This design allows memory mapping and individual tensor loading without reading the entire file, earning its place as a PyTorch Foundation project. Notably, most GPTQ, AWQ, and EXL2 models still utilize .safetensors containers while embedding quantization data inside tensor contents and configuration files.
Choosing the correct model format is critical for modern AI engineering, as developers must continuously balance inference speed, VRAM consumption, and output accuracy. Understanding these architectural differences ensures optimal deployment across hardware-constrained environments.
GGUF emerged as a binary format created by Georgi Gerganov, head of llama.cpp, launched in August 2023 to succeed the legacy GGML format. Unlike its predecessors that broke compatibility whenever hyperparameters changed, GGUF adopted typed key-value metadata to allow seamless additions of new fields without invalidating older files.
Meanwhile, GPTQ—developed by researchers from IST Austria and ETH Zurich—operates as a one-shot, post-training weight quantization technique utilizing second-order Hessian information for rounding adjustments. On the other hand, AWQ from Song Han’s MIT group identifies roughly 1% of salient weights through activation magnitudes rather than weight values, sharply cutting quantization error without requiring model retraining.
"Not all weights matter equally. Protecting roughly 1% of 'salient' weights sharply reduces quantization error."
Song Han's group at MIT
Rounding out the ecosystem is EXL2, the native format for turboderp's ExLlamaV2 inference library built for consumer GPUs. It supports versatile quantization from 2 to 8 bits and paves the way for EXL3, which integrates QTIP technology from Cornell RelaxML to further optimize tensor regularization and packing.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment