Colibri: A Single C File Runs a 744B MoE on 25GB of RAM by Treating Every Expert as Data to Be Staged, Not State to Be Held

Colibri (JustVugg/colibri, Apache 2.0, 17.9k stars) proves the opposite: a 744B-parameter model activates only ~40B parameters per token, and only ~11GB of those change from token to token. The rest is cold data. Colibri turns that observation into a runtime: a single C file, zero dependencies, that streams experts from disk as they are needed, learns which ones are hot, and pins them automatically. The result is a 744B model running on a 25GB laptop at 0.05–0.1 tok/s, a 6× RTX 5090 cluster at 5.8–6.8 tok/s, and everything in between, same engine, same int4 container, same router semantics regardless of where the expert answers from.

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | July 29, 2026

Colibri runs GLM-5.2 (744B total parameters, 19,456 routed experts) on consumer hardware by implementing a three-tier memory hierarchy: VRAM for hot experts, RAM for warm experts, NVMe for cold experts. The dense part of the model (attention, shared experts, embeddings, ~17B parameters) stays resident in RAM at int4 (~9.9 GB). The 19,456 routed experts (~19 MB each at int4, ~370 GB total) live on disk and stream on demand via a per-layer LRU cache with a learned pinned hot-store.

The research paper in the inputs (arXiv:2605.11537, Iowa State University, May 2026) is the academic complement: it proves that MoE expert selection is predictable enough to prefetch with high accuracy using an SRU predictor, and that dynamically replicating overloaded experts achieves near-100% GPU utilization with up to 3x inference speedup on Switch-base-128 while preserving 90–95% of baseline accuracy. Colibri implements the same intuitions (predictive prefetch, learned hot-store) in production C code against a real 744B model.

Scope covered: the per-token execution path, the three-tier memory hierarchy, the PILOT lookahead prefetcher and its 71.6% one-layer-ahead routing predictability, the dual-SSD streaming strategy, MTP speculative decoding mechanics and the int8 head requirement, KV cache compression (57× smaller via MLA), and the MoE-MPMC paper's SRU predictor and expert replication approach. Excluded: the Tauri desktop shell, GLM-5.2's training architecture details, and the Apple Silicon Metal backend (experimental at time of writing).

GLM-5.2's 744B parameters break into two fundamentally different residency categories:

Caption: The five-step per-token path is the key design. ROUTE runs one layer ahead so PLACE has time to fetch the next expert before it is needed. The INVARIANT at the bottom is the correctness guarantee: the engine never silently changes precision to make a placement decision work.

Caption: Each flag is a measurable speedup, not a configuration preference. The +34% from DIRECT=1 is measured on Blackwell hardware. The 71.6% PILOT predictability is measured on routing traces. The 5.8–6.8 tok/s on 6× RTX 5090 is from the experiment log.

Caption: The int4 MTP head failure is silent. There is no error. Acceptance rate just stays near zero. The file size check is the only way to verify before running. This is documented in issue #8 with the full forensic history — one of the most valuable issues in the repo.

Input: Single user query on a 128 GB CPU-only desktop (no GPU). GLM-5.2 int4 with int8 MTP heads, PIPE=1, PILOT=1, model on fast NVMe.

Step 1: Engine startup

Step 2: First query (cold start, no .coli_usage history)

Step 3: Same query, warm (after one run)

Step 4: Output (abbreviated)

The numbers across hardware classes:

The invariant is the design. Colibri's README states it explicitly and the codebase enforces it: "Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics." This means the model answer from a 25 GB laptop running experts from disk is identical to the model answer from a 6× RTX 5090 with full VRAM residency. Only latency changes. Not quality. Not routing.

This is a hard constraint to maintain. It requires that every tier in the memory hierarchy serves the same weight format (int4), that the router never sees different inputs based on placement decisions, and that KV cache semantics are preserved across restarts. The KV cache compression (MLA attention, 576 floats per token instead of 32,768, a 57× reduction) and its persistence across restarts (.coli_kv) mean a conversation can resume exactly where it left off, byte-identical to an uninterrupted session.

What Colibri trades away is throughput predictability. The engine is disk-bound on most consumer machines. NVMe read bandwidth, not compute, determines decode speed when experts are cold. A system that hits a 9 GB/s NVMe drive decodes faster than one hitting a 2 GB/s DRAM-less QLC drive, on identical hardware. Teams building latency-sensitive applications on Colibri need to measure their specific storage hardware, not just their GPU.

The learning cache (.coli_usage, updated every turn) is the mechanism that converts a slow first run into a fast steady state. But it also means that workload diversity hurts performance. A chatbot serving 100 different topic domains will have a more diffuse routing heat distribution than a coding assistant that routes heavily to the same subset of experts. The same model on the same hardware can decode at 1.8 tok/s for one workload and 0.6 tok/s for another, depending entirely on expert cache temperature.

The expert atlas (19,456 experts characterized by topic affinity) is the moat nobody is discussing. The Brain and Atlas visualizations in Colibri's dashboard are not cosmetic. They represent a completed measurement campaign across all 19,456 experts in GLM-5.2, identifying which experts specialize in poetry, law, Chinese, SQL, mathematics, and other domains. This measurement (issue #175 in the repo, "measured expert atlas") is the foundation for the intelligent hot-store: rather than pinning randomly hot experts, a topic-aware workload can pin the expert cluster aligned to its domain. Reproducing this for a different MoE model requires running the entire measurement campaign, which requires the model weights, compute time, and the measurement infrastructure.

The MoE-MPMC paper's insight validates Colibri's approach from a different direction. The paper demonstrates that switch-base-128 with 1.3% GPU utilization (baseline) jumps to 82% with dynamic expert replication. Colibri does not run on the same hardware class as the paper's experiments (A100 80 GB HPC cluster), but the underlying premise is identical: routing has measurable structure, structure enables prediction, and prediction enables pre-placement. The 71.6% one-layer-ahead routing predictability in Colibri is the same finding quantified differently.

The single-file C engine (c/glm.c) is a deliberate moat against complexity. The README states the project vision explicitly: "the engine is deliberately small enough that the next optimization can come from you." Every optimization in Colibri (PIPE, PILOT, DIRECT, dual-SSD, CUDA tier) came from community members measuring on their own hardware and contributing back. A 40,000-line Python codebase with external framework dependencies does not have this property. The C single-file design is what makes the contribution loop tight enough to actually function.

Insight One: Colibri's 0.05–0.1 tok/s baseline is the honest number the community keeps hiding. Every MoE offloading project reports numbers from the best hardware configuration. Colibri's README leads with the 25 GB laptop baseline: 0.05–0.1 tok/s cold, the "proven floor where this project started." At that speed, a 200-token response takes 30–60 minutes. That is not a usable inference server. It is a research tool and a proof of concept. The useful deployment range starts at a 128 GB CPU desktop (~1.8 tok/s) and becomes genuinely practical at a single consumer GPU. Teams reading the headline "runs 744B on 25 GB" and expecting interactive latency are going to be disappointed. The hardware requirement for practical use is closer to 128 GB RAM or one RTX 5090, neither of which is a consumer machine in any meaningful sense.

Insight Two: The MoE-MPMC paper's 3x speedup headline conflates GPU-bound and disk-bound inference. The paper demonstrates 3x speedup on a single A100 80 GB GPU where all experts fit in VRAM. In that setting, the bottleneck is GPU utilization (1.3% baseline for switch-base-128 because most experts sit idle), and replication addresses it directly. Colibri's bottleneck on a 25 GB machine is NVMe bandwidth, not GPU utilization. Expert replication does not help when the constraint is I/O throughput, not compute parallelism. The paper and the runtime solve different problems in the MoE inference space, which makes them complementary rather than competitive, but the 3x speedup number does not translate to the disk-bound regime.

Colibri literally gets faster the more you use it, and that improvement persists across restarts. The .coli_usage file records routing heat from every conversation turn. On next startup, the engine reads this file and pins the hottest experts into RAM before any request arrives. A coding assistant that runs 50 sessions accumulates 50 sessions of routing evidence. Its warm startup is faster than its first run by an amount that depends on how consistent the workload has been. This is not a cache in the conventional sense. It is a learned placement policy that improves monotonically with usage. The analogy in the README is correct: it is a JIT compiler that compiles the hot paths, except the "hot paths" are expert weight locations, and the "compilation" is placement in the memory hierarchy. Workload-specific deployment (coding assistant, legal assistant, medical Q&A) gets measurably better inference performance over time than generic deployment, on identical hardware.

Modern AI models have become so large that they no longer fit in the memory of any single machine. A 744-billion-parameter model, if you tried to load it all at once, would require roughly 370 gigabytes of fast GPU memory, which does not exist in any affordable hardware today.

But here is the thing: these large models are not actually using all 744 billion parameters on every word they generate. They use a technique called Mixture of Experts, where the model is divided into thousands of small specialist subnetworks. For each word, only a tiny fraction of those specialists (about 40 billion parameters) actually do any work. The rest sit idle.

Colibri exploits this fact. Instead of trying to hold the entire model in memory, it keeps only the always-active parts in RAM (about 10 gigabytes) and stores the specialist subnetworks on disk. When the model needs a specialist to process a word, Colibri reads it from disk, uses it, and then moves on. To make this fast, it learns from experience which specialists get used most often and keeps those in faster memory automatically.

The analogy is a librarian who, instead of memorizing an entire library, learns which books get checked out most often and keeps those on a small desk for quick access. The rest stay on shelves, but the librarian is fast enough that it rarely matters.

This matters because it moves the constraint from "how much memory you have" to "how fast your storage is." That is a much cheaper problem to solve, and it means genuinely frontier-scale models can run on hardware people actually own.

Colibri is not a distributed inference server. It is not a production serving stack. It is a proof that a 744B-parameter model is a data staging problem, not a memory problem, and that staging can be learned, automated, and improved continuously through use.

The MoE-MPMC paper (arXiv:2605.11537) provides the theoretical foundation: routing is 71.6% predictable one layer ahead, expert activation is structurally biased toward a small fraction of the expert pool, and dynamic replication can push GPU utilization from 1.3% to 82%. Colibri implements those insights in a single C file with no external dependencies, on real hardware, against a real 744B model.

The community benchmark loop, where someone measures something on their hardware and opens an issue with numbers, then someone optimizes and measures again, is how this engine actually works. At 17.9k stars and 1.7k forks in a field where most inference projects require multi-GPU datacenter hardware to matter, that loop is working.

Sign up for a free trial of Attio, the agentic CRM.

Ask Attio to build a daily workflow that surfaces the deals that need your attention today, like anything with a stage change, a recent reply, or a new signal in the last 24 hours.

Review your pipeline in Claude, synced live from Attio via MCP.

That's it.

Try Attio Now