Run a 2.78 trillion parameter model in 8GB (Kimi K3 setup)

Someone wrote an inference engine for Kimi K3, 2.78 trillion parameters, in portable C99. No GPU, no BLAS, no framework, seven C files. It runs on one ordinary CPU in 8.24 GB of RAM.

This is the measured run from the README, on the laptop preset:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
           --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

Correct, in 261 seconds, at about 33 seconds a token, inside 8.24 GB. It is a base model with no chat template, so it continues rather than replies: after answering it simply carried on, and what came next happens to look like the next line of a JSON file. That is what a base model does when nobody wrapped it in a chat format.

You can run the engine today without the 1.7 TB

This is the part worth knowing, because almost nobody has 1.7 TB spare. The whole test suite runs with no checkpoint, no network and no Python, in about a minute.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

make -j            # seconds. seven C files, a compiler and OpenMP
make test          # under a minute

It ends like this, or it failed:

GATE 1  teacher forcing : 32/32 positions match tf_pred
        generated span  : 20/20  <- must be exact
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental    : 20/20 generated tokens match full_ids

VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY

That is every kernel, the streaming cache, the safetensors reader, the config reader and the tokenizer, checked end to end against a PyTorch reference from fixtures committed to the repo. You get to see the thing prove itself without downloading a terabyte.

What it needs if you do want the real model

OS	Linux, x86-64
CPU	AVX2 + FMA. AVX-512 not needed
RAM	8 GB and up
Storage	about 1.7 TB free: 1.56 TB checkpoint plus a 109 GB packed trunk
Toolchain	GCC 9+ or Clang 10+
Python	3.9+, for the download and pack tools only

Run ./scripts/k3-doctor.sh at any point. It checks your toolchain, sizes your RAM to a preset, measures your storage, and prints the exact command to run next.

More RAM only buys you clock

laptop        3.0 / 1.0      8.2 GB peak RSS.  the floor. runs, slowly
desktop      16.0 / 10.0    31.9 GB peak RSS
workstation  60.0 / 30.0    95.5 GB peak RSS
server      110.0 / 13.0    ~128 GB peak RSS.  fastest
max         110.0 / 109.0   ~224 GB peak RSS

The honest finding, and it is the author's own: 28 times the memory buys about 1.7 times the speed, and the output is byte-identical from the smallest machine to the largest. Only the clock changes. 8 GB gets you roughly 27 to 33 seconds per token, a 192 GB box gets you about 6.

That is the real result here. Memory was never buying a better answer, only a faster one.

Where it will disappoint you

The storage gate is brutal and there is no way around it. About 1.7 TB free, ideally on fast local disk. This is not a clear-some-space problem. If you do not have a spare drive, run the test suite above and enjoy it for what it is.

Linux x86-64 only. It uses O_DIRECT, posix_memalign and getrusage. No macOS, no Windows.

Half a minute per token at the floor. A one sentence answer takes minutes, not seconds. This is a demonstration that the arithmetic fits, not a chatbot you will use daily.

No chat template. It continues text, it does not reply. Ask it a question and you get a plausible continuation of a document containing that question.

Source

github.com/FareedKhan-dev/kimi-k3-in-c, Apache 2.0, 6,040 stars. Every figure in its README comes from measurement output committed under docs/data/.

---

How much RAM and free disk have you got?

Ask me in The Rabbit Hole and I will tell you whether to run the real thing or just the test suite.

It is the room where I keep the parts that don't fit on a page like this: the full run of numbers, what broke the first time, and the ones that looked good and then didn't work at all.

Two catches, up front, because that's the rule here. It's private, so you have to be inside to read anything. And it's new, so you'd be early rather than joining something busy.

Free: skool.com/the-rabbit-hole-3206