Pick the right GGUF for your RAM (the quant picker)

What it is. GGUF is the model file format llama.cpp reads, and Ollama, LM Studio and most local AI apps run on llama.cpp underneath. One model ships as a dozen GGUF files; the letters in the name (Q4_K_M, Q8_0, IQ4_XS) say how many bits each weight was squeezed to. Same model, different file sizes, different quality. This page is how to pick one, and how to size the part of the RAM bill the file size does not show you.

Every number on this page is from llama.cpp's own tools/quantize/README.md, the bartowski and unsloth GGUF repos on Hugging Face, and the model's config.json, all read 16 Sep 2026. Worked example: Llama 3.1 8B Instruct, because it is the one most people try first.

The files, Llama 3.1 8B Instruct

File	Size	Real bits per weight	bartowski's note
BF16	16.07 GB	16.0	the original, no loss
Q8_0	8.54 GB	8.50	extremely high quality, generally unneeded
Q6_K	6.60 GB	6.56	very high quality, near perfect, recommended
Q5_K_M	5.73 GB	5.70	high quality, recommended
Q4_K_M	4.92 GB	4.89	good quality, default size for most use cases, recommended
IQ4_XS	4.45 GB	4.46	decent quality, smaller than Q4_K_S
Q3_K_M	4.02 GB	4.00	lower quality but usable
Q2_K	3.18 GB	3.16	very low quality but surprisingly usable

Notice the third column. "4-bit" Q4_K_M is really 4.89 bits per weight, because the scales and a few important tensors keep more precision. That is why 16 GB becomes 4.9 GB and not 4 GB: a 3.3x cut, not 4x.

Step 1: how big a file can you run

Two rules, both from bartowski's model card, and they are the ones everyone uses:

1. Fastest: the whole file fits in your GPU's VRAM. Pick a file 1 to 2 GB smaller than the VRAM.
2. Best quality that still runs: add system RAM and VRAM together, pick a file 1 to 2 GB smaller than that total. On a Mac, unified memory is the total.

If you do not want to think about it, take a K-quant (QX_K_M). Below Q4 on an Nvidia or AMD card, the I-quants (IQ3_M, IQ4_XS) are better for their size; on CPU and Apple Metal they run slower than the K-quant of the same size.

Step 2: the part the file size hides

The file is the weights. The context (everything the model is currently reading) lives in the KV cache, and llama.cpp keeps that cache in 16-bit by default, whatever quant you picked for the weights.

For Llama 3.1 8B: 32 layers x 8 KV heads x 128 head dim x 2 (K and V) x 2 bytes = 128 KB per token.

Context	KV cache at 16-bit	with -ctk q8_0 -ctv q8_0
4,096 tokens	0.5 GB	about 0.27 GB
8,192 tokens	1.1 GB	about 0.55 GB
32,768 tokens	4.3 GB	about 2.2 GB
131,072 tokens (the model's full window)	17.2 GB	about 8.6 GB

At the full window the cache alone is bigger than the 16-bit weights you just shrank. So the honest budget is: file size + cache for the context you actually use + headroom for the OS. A 4.9 GB file with 8k of context wants about 6 GB, not 4.9.

The same arithmetic for any other model: open its config.json, take num_hidden_layers x num_key_value_heads x (hidden_size / num_attention_heads) x 4 bytes.

Step 3: run it with llama.cpp

brew install llama.cpp

(Homebrew on macOS or Linux. Everywhere else, the release binaries are on the llama.cpp GitHub releases page.)

llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -c 8192 -fa on -ctk q8_0 -ctv q8_0

-hf downloads the file straight from Hugging Face on first run (Q4_K_M is its default if you leave the suffix off). -c 8192 is the context size, which is what the cache table above is priced on. -fa on turns on flash attention, which llama.cpp requires before it will quantize the V cache. Then open http://127.0.0.1:8080 in a browser: the server ships its own chat page and an OpenAI-compatible API.

To see what it actually costs: the server log prints the KV cache size at startup. Change -c and watch it move.

Step 3b: or with Ollama

ollama run llama3.1:8b

That tag is a 4.9 GB file, the Q4_K_M. For a smaller cache, set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 in the environment before starting the Ollama server; the default is f16.

What 4-bit saves and what it does not

llama.cpp's own table for this model, one machine (it does not name which):

• Weights: 14.96 GiB at F16 to 4.58 GiB at Q4_K_M. 3.3x.
• Text generation: about 2.5x faster, because decoding is bound by how many bytes get read per token.
• Prompt processing: slightly slower, 923 to 822 tokens a second. Quantization does not make reading your prompt faster.
• The KV cache: unchanged unless you change it. See Step 2.
• Quality: some loss, measured as perplexity and KL divergence, and it grows as the bits drop. The README gives no single number for this model, so test the quant on your own prompts before trusting it, and keep anything byte-exact (IDs, hashes, code you will paste back) at a higher quant or check it by hand.

Where this lives

The Rabbit Hole is free to join. The course library is free; optional Premium adds weekly AI Builder Office Hours: https://www.skool.com/the-rabbit-hole-3206/about