Run a 744B model in 9.9GB of RAM (Colibri setup)

Colibri is a pure C engine that runs frontier Mixture-of-Experts models on ordinary machines. It keeps the small dense core of the model resident in RAM and streams the routed expert weights off your disk as each token needs them. That is the whole trick, and it is why a 744B model has only 9.9GB of weights resident.

GLM-5.2 is the reference model. Five more families run on the same engine, and none of them needs a GPU.

Model	Total / active	Disk for the weights	RAM	GPU
OLMoE	7B / 1B	~7 GB	8 GB	not needed
Qwen3.6-35B-A3B	35B / 3B	~20 GB	24 GB	optional, 7x faster on two 8 GB cards
DeepSeek V4 Flash	284B / 13B	~167 GB	16 GB min, 32 GB comfortable	optional
GLM-5.2 (this page)	744B / 40B	~372 GB	16 GB min, 24 GB comfortable	not needed
Inkling	975B / 41B	~469 GB	25 GB with the int4 dense container	not needed
Kimi K3	2.8T / 104B	~1.6 TB	32 GB+	not needed

A GPU only ever makes it faster. Speed is set by your disk, because that is where the experts are read from on every token.

Install

A prebuilt release is the fastest path. Linux, macOS and Windows, no compiler needed. Take the archive for your platform from Releases:

mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info                         # engine ready

You get the engine, the coli launcher and its Python helpers. Nothing to rename or configure, coli finds the engine next to itself. You only need Python 3 installed: the launcher and the API gateway are Python, while the engine itself is pure C with zero dependencies.

Or build from source, which needs gcc or clang with OpenMP:

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh                                # checks gcc/OpenMP, builds, self-tests

Get the model

A pre-converted GLM-5.2 int4 container is on Hugging Face. It is about 372 GB, so put it on a disk with the room, ideally a fast one:

huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp

Pull the gs64 container, not the older per-row int4 mirrors. The per-row ones measure about 9 percentage points worse on quality and are the root cause of the think-mode loops and never-terminating generations people report. The MTP head must be int8, not int4, because int4 gives 0% draft acceptance. Check with ls -l <model>/out-mtp-*: int8 is the one at 3527131672 / 5366238584 / 1065950496.

Or convert from the FP8 source yourself, one resumable command that never needs the full 756 GB on disk at once:

./coli convert --model /nvme/glm52_i4   # downloads and converts shard by shard, one time

Run it

COLI_MODEL=/nvme/glm52_i4 ./coli doctor   # read-only readiness check, run this first
COLI_MODEL=/nvme/glm52_i4 ./coli plan     # inspect the planned VRAM/RAM/disk placement
COLI_MODEL=/nvme/glm52_i4 ./coli tune     # measure and save this machine's fastest safe profile
COLI_MODEL=/nvme/glm52_i4 ./coli chat     # RAM budget, cache and MTP auto-detected
./coli web --model /nvme/glm52_i4         # API plus dashboard, opens a browser

On Windows the same commands work with python coli chat --model D:\glm52_i4.

Run doctor and tune before you judge the speed. tune measures your actual machine and saves the fastest safe execution profile, and on a disk-bound engine that is not a cosmetic difference.

If 372GB is too much, start smaller

Same engine, same commands, nothing about the command line changes between models. Point COLI_MODEL at a different directory.

OLMoE is about 7 GB on disk and wants 8 GB of RAM. That is the one to try first if you just want to see the streaming engine work on a machine you already have.

Qwen3.6-35B-A3B is about 20 GB on disk and wants 24 GB of RAM, and it is the one that benefits most from a GPU: the VRAM expert tier measured 1.44 to 10.05 tok/s on two 8 GB cards, with output bit-identical to the CPU path. Pull the int4-gs64 container.

Where it will disappoint you

You are downloading 372 GB before anything happens. This is the part the headline number hides. The model does not fit in 9.9GB, it runs with 9.9GB of weights resident. The other 19,456 routed experts live on your SSD and get read from there on every single token.

9.9GB is the resident weights, not your RAM requirement. The machine still wants 16 GB minimum and 24 GB to be comfortable. Those are two different numbers and it is easy to read the first as the second.

It is slow on hardware you already own, and how slow depends entirely on your disk. The project publishes the range honestly. On a 128 GB CPU-only desktop it is about 1.8 tok/s warm. On a single RTX 5070 Ti it is 1.07 tok/s. On a 25 GB dev box it is 0.05 to 0.1 tok/s cold, which the README itself calls "the proven floor where this project started, and still the honest baseline". That last one is one token every ten to twenty seconds.

The screenshot everyone shares is running on six RTX 5090s. The repo's own description says "run frontier MoE models on hardware you already own", and the dashboard image behind that headline is a full-residency box with six cards in it, benchmarked at 5.8 to 6.8 tok/s. Both things are true at once. Just know which one you are signing up for before the 372 GB starts.

A model bigger than your free RAM inherits the previous run's page cache. So the same command run twice can give you very different speeds, and the second number is not the honest one. This bit anyone benchmarking this class of engine, and it is worth knowing before you post your own figure.

Source

github.com/JustVugg/colibri. Pure C, zero engine dependencies, 25,962 stars as of 24 August 2026. Quality is measured rather than assumed: the int4 container's quantization cost and the scale-granularity ablations are published in the repo's own benchmark docs, alongside the misses.

How much free disk, how much RAM, and is that disk NVMe or SATA?

Ask me in The Rabbit Hole and I will tell you which model to pull and roughly what speed to expect.

It is the room where I keep the parts that don't fit on a page like this: the full run of numbers, what broke the first time, and the ones that looked good and then didn't work at all.

Two catches, up front, because that's the rule here. It's private, so you have to be inside to read anything. And it's new, so you'd be early rather than joining something busy.

Free: skool.com/the-rabbit-hole-3206