Sponsored by
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | August 24, 2026
The Promise
Two measured bandwidth numbers, taken on your own machine, decide where every missed expert runs. That single ratio is the whole system.
What this covers: the expert memory hierarchy, the q* split policy, why it has to live inside a CUDA Graph, the cache locality numbers, and the honest limits.
What this excludes: MoE training, router design, quantization math, and the desktop GUI. This is about serving.
What It Actually Does
FreeToken is an edge-native MoE serving engine from a Berkeley, MIT and UT Austin group including Shuo Yang, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu and Ion Stoica. It went up on arXiv on August 17, 2026, paper 2608.16157, with an Apache 2.0 repo at FlashML-org/FreeToken and a desktop build at flashml.ai.
Strip the framing and the function is narrow: it keeps the complete routed-expert pool in host RAM as the source of truth, treats all remaining VRAM as one global LRU cache of complete (layer, expert) slots, and decides per decode step which cache misses cross PCIe and which get computed in place on the CPU.
The numbers it is claiming, on an RTX 5090:
Metric | FreeToken | Best baseline |
|---|---|---|
Qwen3.6-35B-A3B decode | 77 to 83 tok/s | 1.8x to 2.3x lower |
DeepSeek-V4-Flash decode | 22 to 25 tok/s | 1.5x to 1.9x lower |
Worst case TTFT, any workload | under 44 s | llama.cpp 232 s, Ollama 179 s, KTransformers 946 s |
Decode drift, single turn to agentic | within 12% | KTransformers loses 31% by the second workload |
And at the extremes: a 753B GLM-5.2 at 14.9 tok/s on one RTX PRO 6000 against llama.cpp's 7.3, and 39.3 tok/s for a 35B model on an 8 GB RTX 4060 laptop. The production median decode speed of Codex, measured on real traces, is 33 tok/s. The laptop beats it.
No approximation is involved. GPU and CPU partial sums merge exactly, so the routed computation is bit-faithful to the checkpoint. That separates it cleanly from HOBBIT, SiDA and SMoE, which buy bandwidth by degrading fidelity.
The Architecture, Unpacked
Caption: Focus on the two arrows leaving the router. Every prior offloading system has only the left one. The right one, computing a missing expert where it already lives, is the entire contribution.
Three decisions carry this design, ranked.
One, the slot is a complete expert, not a tensor shard. Every bank is indexed by the flattened identifier lE + e, so residency, lookup, eviction and execution all speak the same logical name. GPU kernels and the CPU executor share one identity for an expert regardless of its physical quantization layout. Without this, a single fused copy across all banks would be impossible.
Two, prefill and decode share one slot pool. There is no separate prefill cache and no handoff between phases. Experts that survive prefill seed the latency-sensitive decode phase for free. When the pool cannot spare two full layers, FreeToken degrades to on-demand loading instead of oversubscribing VRAM.
Three, all routing-dependent control lives on the device. One kernel per MoE layer deduplicates routed experts, classifies them against the residency table, derives q, picks eviction victims and rewrites logical expert IDs into physical slot IDs or a CPU-assignment flag. Victim selection is a single pass that surfaces the K least recently used slots at once, so the cost is one pass regardless of how many misses actually occurred.
That last point is the one to sit with. A host-controlled cache would force a device synchronization at every MoE layer. Fixed cost, every layer, every token.
The Locality Result Nobody Is Quoting
The paper replays identical routing traces from all four workloads against three placement policies at equal cache capacity. At what an RTX 5090 can actually hold:
Cache capacity | FreeToken global LRU | KTransformers | llama.cpp |
|---|---|---|---|
37% of Qwen3.6 expert pool | 16% miss | 41% miss | 62% miss |
11% of DSV4-Flash pool | 39% miss | 59% miss | 89% miss |
Now do the arithmetic the paper leaves on the table. If placement is uncorrelated with routing, expected miss rate is simply 1 - capacity. At 37% capacity that predicts 63%. llama.cpp measures 62%. At 11% capacity it predicts 89%. llama.cpp measures 89%.
llama.cpp's routing-blind static split performs at chance, on both models, to within one point. It is not a weak cache. It is not a cache.
FreeToken converts 11% of DSV4's expert pool into a 61% hit rate. That is a 5.5x lift over what capacity alone buys.
The Code, Annotated
Calibration decides the backend, not the GPU
# Run ONCE per machine. This measures the real cpu and offload MoE kernels
# on the deployed tensor shapes, not vendor spec-sheet numbers.
ft bench bw --dtype nvfp4,bf16
# Writes ~/.cache/freetoken/benchbw.json, keyed on expert format + GPU name,
# so a profile copied from different hardware is IGNORED, not misapplied.
# Decision rule: --threshold 2.0 → recommend hybrid when Bh > 2 x Bp
# ← THIS is the trick
ft serve --model ~/models/Qwen3.6-35B-A3B
# --model is the ONLY required flag. dtype, attention backend, MoE backend,
# cache sizes, CUDA-graph sizes and the tool-call parser all resolve from
# the checkpoint and the GPU. Ready at 127.0.0.1:1919.
Caption: The threshold is 2.0, not 1.0. FreeToken will not enable CPU co-execution just because it would help slightly. That gap is deliberate and Insight Two explains it.
Run that threshold against the paper's own six machines and the backend selection inverts the intuition that a better GPU means more GPU work:
System | Bp (GB/s) | Bh (GB/s) | Bh / Bp | Backend chosen | Misses sent to CPU |
|---|---|---|---|---|---|
RTX 5090 desktop | 49.0 | 53.8 | 1.10 | offload | 9% |
RTX 5090 server | 52.7 | 77.3 | 1.47 | offload | 32% |
RTX 3090 | 25.3 | 56.7 | 2.24 | hybrid | 55% |
RTX 4090 | 25.1 | 63.2 | 2.52 | hybrid | 60% |
RTX PRO 6000 | 51.5 | 178 | 3.46 | hybrid | 71% |
RTX 4060 laptop | 11.8 | 47.5 | 4.03 | hybrid | 75% |
The flagship desktop barely uses the CPU. The cheap laptop leans on it for three quarters of its misses. What FreeToken adapts to is the PCIe link, not the silicon.
The split policy itself
# Faithful reconstruction of the q* policy from Equations 1-4 and Section 4.1.
# In the shipped engine this is a CUDA kernel, not Python, and that is the point.
def split_misses(missing_experts, B_P, B_H):
"""Divide this step's cache misses between PCIe fill and in-place CPU exec."""
m = len(missing_experts)
# Both branches read the SAME host memory subsystem. A saturated PCIe
# transfer leaves only B_H - B_P for the CPU. Balancing the two concurrent
# branch times, qS/B_P == (m-q)S/(B_H - B_P), collapses to this:
q = round(m * B_P / B_H) # ← THIS is the trick: expert size S cancels
# Never stop warming the cache, even when the CPU is doing almost all the
# work. On a 4060 laptop q* rounds to ~0.99 for m=4; this clamp is what
# actually earns the throughput. See Insight Two.
q = max(1, min(q, m))
fill_set = missing_experts[:q] # PCIe → slot, GPU executes, STAYS resident
cpu_set = missing_experts[q:] # executes where it already lives
return fill_set, cpu_set
# RTX 4060 laptop, m = 4: q* = round(4 * 11.8 / 47.5) = round(0.99) = 1 → 1 fill, 3 CPU
# RTX 5090 server, m = 4: q* = round(4 * 52.7 / 77.3) = round(2.73) = 3 → 3 fill, 1 CPU
Caption: Expert size S cancels out of the ratio. The policy needs two scalars and one multiply, which is precisely why it can live inside a statically captured graph.
Note the execution order in the real engine: the CPU branch launches first, then the GPU miss path runs its cache update, batched copy and grouped evaluation. Exposed layer latency is the slower of the two, which is exactly the quantity the equation balances.
Resizing the cache without a restart
ft ctl cache # current pool table
ft ctl cache --moe 512 --wait 300 # rebuild the expert cache to 512 slots
ft ctl cache --kv 64k # or shift the budget toward KV tokens
# Legal because the host-resident pool is the source of truth: shrinking the
# GPU cache costs throughput and NOTHING else. No restart, no weight reload.
Caption: This exists because a browser or a game can claim gigabytes of VRAM mid session. Correctness is decoupled from residency, so memory becomes a tuning knob rather than a constraint.
It In Action
Input: a SWE-bench repository issue driven through the OpenCode harness, on the RTX 4060 laptop. 8 GB VRAM, PCIe 4.0 x8, Core i9-13900H, 32 GiB LPDDR5. Model is Qwen3.6-35B-A3B in NVFP4.
Step one, calibrate. ft bench bw measures Bp at 11.8 GB/s and Bh at 47.5 GB/s. Ratio 4.03, over the 2.0 threshold, so the profile recommends hybrid.
Step two, launch. ft serve reserves a floor of 8,192 KV tokens, then fills the remaining VRAM with expert slots. Expert weights load from disk straight into their final host bank layout and are pinned only after the banks are full. Pinning empty buffers first would fault in and zero gigabytes of pages just to overwrite them.
Step three, prefill. Chunked at 8,192 tokens with the two-buffer overlap on. Layer l+1 streams while layer l computes.
Step four, one decode layer. The router selects experts, the lookup kernel finds four unique misses. q* = 4 x 11.8 / 47.5 = 0.99, rounds to one.
F = 1 expert → PCIe fill T_fill = 1S / 11.8 = 0.0847 S
C = 3 experts → CPU in place T_cpu = 3S / (47.5 - 11.8) = 0.0840 S
exposed = max(both) = 0.0847 S
naive, all four over PCIe: 4S / 11.8 = 0.3390 S (4.0x worse)
Step five, output. 39.3 tok/s sustained, or 25.4 ms per token. That is 92% of the RTX 4090's rate on the same workload, and 1.8x the strongest baseline on that laptop.
For contrast, the same pipeline on the 5090 server prefills 8,192 tokens in 1.19 to 1.22 s. Stream 64.4 GB once at 52.7 GB/s and you get 1.222 s. Prefill is not compute bound, it is pinned to the PCIe 5.0 x16 ceiling, and throughput climbs to 6.7k tok/s at 16k tokens. Disable the second buffer and it costs 19% at 4k, 25% at 8k, 26% at 16k. The penalty grows with prompt length because there is more computation available to hide behind.
Why This Design Works, And What It Trades Away
It works because it attacks the right variable. Every offloading system before it treated a missing expert as bytes that must move. FreeToken treats it as work that can execute wherever it already sits. Once you accept that framing, the closed-form ratio falls out of a residual-bandwidth argument in four lines, and the LRU cache and the prefill pipeline are just competent engineering around it.
The tail latency result is the one that matters operationally. Worst case TTFT under 44 s everywhere, against llama.cpp at 232 s, Ollama at 179 s and KTransformers at 946 s. OpenClaw ships a 120 s idle watchdog. Claude Code's default request timeout is roughly ten minutes. Tail TTFT is not a latency statistic on agentic workloads, it is a binary availability gate. An engine that averages well and spikes to 946 s does not serve the session at all.
Now read the benchmark honestly. The two RTX 5090 columns share identical GPU silicon and differ only in the host. Moving from the many-channel server to a dual-channel consumer desktop costs FreeToken 4% of decode rate. llama.cpp keeps 80%. FreeToken's largest consumer win, 2.1x, is on the desktop, and most of that margin is the baseline collapsing rather than FreeToken accelerating. On that machine q* sends only 9% of misses to the CPU. The headline mechanism is barely firing.
What it trades away:
Deployability. Linux x86_64, NVIDIA only, driver r580+ with CUDA 13, kernels JIT compiled on first use. No macOS, no Apple Silicon, no AMD, no Windows for the CLI. llama.cpp's actual reach is not in the comparison set, and for a project whose thesis is "the machines users already own," that is a large asterisk.
Host RAM, not VRAM, is the real gate. The pool must fit in host memory. That is roughly 140 GB for DSV4-Flash and a 433 GB checkpoint for GLM-5.2. The "gaming desktop" running a 284B model carries 192 GB of DDR5. Community reaction has already caught this: the frequent reply to the 5090 demo is disappointment on discovering the RAM requirement.
The bandwidth profile is static. Bp and Bh are measured once at deployment and cached. But the paper's own premise is that edge resources fluctuate. Elastic memory management handles VRAM changing mid session. Nothing re-profiles bandwidth when a game starts saturating your memory controller, which is exactly when q* becomes wrong.
Single user assumptions. --max-running-requests defaults to 4. This is a personal serving engine, not a multi-tenant one.
No independent verification yet. Reproduction trackers currently flag the numbers as not paper-verified. Everything above is the authors' measurement.
Technical Moats
The formula is trivial. Anyone can derive q* = m·Bp/Bh in an afternoon. The moat is that a trivial formula is the only kind that survives CUDA Graph capture, and that constraint is what competitors keep failing.
HybriMoE rebalances CPU and GPU queues by simulating a schedule per step. That simulation is host-side, so it cannot be captured. SMoE uses a greedy two-pointer heuristic, same problem. llama.cpp cannot maintain graph execution in hybrid mode at all. Each of these is a smarter policy that costs more than it saves, because per-token Python or host scheduling reintroduces the synchronization the graph exists to remove. FreeToken picked the dumbest adequate policy specifically so it could be represented as fixed-shape work buffers and device-resident valid counts inside a static graph.
Second, the engineering underneath is not glamorous and takes a long time. Single-pass K-victim LRU selection. A persistent C++ worker pool pinned to physical cores with architecture-specific SIMD and in-kernel dequantization. The FTW format that skips tensor discovery and repacking at launch. Parallel direct I/O into exact-size host banks. A pure-CPU fallback for platforms where pinning is refused by the OS or driver.
Third, ecosystem position. Anthropic and OpenAI compatible endpoints plus ft launch writing provider configs for Claude Code, Codex, OpenCode, OpenClaw and the DeepSeek harness, with cloud API keys stripped from the child environment so the agent cannot silently fall back to a paid endpoint. That detail signals a team that has actually watched this fail.
Fourth, lineage. The substrate is borrowed from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp, with a co-author list that overlaps vLLM and SGLang directly. That is not a moat you copy.
Insights
Insight One: the expert prediction research program was aimed at the wrong variable.
Three years of work went into predicting which experts a token will route to. Mixtral-offloading paired an LRU cache with speculative prefetching. MoE-Infinity traced request-level activation patterns. ProMoE, ExpertFlow and FineMoE each sharpened the predictor further.
FreeToken contributes nothing to prediction. It runs a plain global LRU with no predictor at all, and beats every one of them.
The reason is structural, and the paper states it in one clean sentence in related work: these systems differ in how well they predict misses, not in how they serve them. Every miss still ends as a PCIe transfer, so decode stays bounded by the link no matter how accurate prediction becomes, while host compute sits idle. WiSP's own analysis reaches the same conclusion independently. The evidence is blunt: MoE-Infinity, the sophisticated trace-based prefetcher, manages 8.8 tok/s on the math workload against FreeToken's 77 to 83 on the same model, and cannot serve the other three workloads at all.
A perfect oracle predictor would not have closed that gap. The ceiling was never prediction accuracy.
Insight Two: on bandwidth-poor machines, q* does not make the current token faster. At all.
Work the laptop numbers against a pure-CPU backend, where no PCIe transfer competes for host bandwidth and the CPU gets all 47.5 GB/s:
hybrid, q* = 1: exposed = max(1S/11.8, 3S/35.7) = 0.0847 S
pure CPU, q = 0: exposed = 4S / 47.5 = 0.0842 S ← FASTER
The hybrid split is 0.6% slower on that step. Every current-token benefit of the celebrated policy has vanished, and it vanishes precisely on the hardware the system is marketed for, because when Bp is small relative to Bh the residual Bh - Bp is nearly all of Bh anyway.
So where does the laptop's 1.8x come from? The clamp. q is floored at one so the cache keeps warming even when the CPU handles nearly everything. That one resident expert per layer per step is not about this token, it is about the hit rate of the next thousand. The mechanism the paper names its policy after balances the current step, but the throughput on bandwidth-poor hardware is earned by a one-line guard against it.
This also explains the otherwise arbitrary --threshold 2.0. Below roughly 2x, the CPU branch is not worth the coordination. Above it, you are really buying cache warmth, not parallelism.
Takeaway
An 8 GB RTX 4060 laptop on a PCIe x8 link sustains 39.3 tok/s, which is 92% of what a 24 GB RTX 4090 on a x16 link delivers on the same model and workload. Three times the VRAM and twice the PCIe bandwidth buy 8%.
The binding constraint is host-side expert bandwidth, and those two machines are only 1.33x apart there, 47.5 against 63.2 GB/s. VRAM capacity buys hit rate with sharply diminishing returns once locality is being exploited properly. If you have been sizing local inference machines by VRAM, you have been optimizing the wrong line item. Check your DRAM channels first.
Second order effect worth pricing: 39.3 tok/s on a thin laptop exceeds the 33 tok/s median decode speed of Codex measured in production traces. The local option is no longer the slow option.
TL;DR For Engineers
A global LRU holding 37% of Qwen3.6's expert pool misses 16% of decode reads. llama.cpp's static split misses 62%, which is exactly what capacity alone predicts with zero locality exploited.
q* = m·Bp/Bhruns as a CUDA kernel inside a captured graph. It is trivial on purpose, because host-side scheduling cannot be captured and costs more than it saves.Prefill is transfer bound by design: 8,192 tokens in 1.19 to 1.22 s, which is 64.4 GB at 52.7 GB/s. Disable double buffering and lose 26% at 16k.
Tail TTFT stays under 44 s against llama.cpp's 232 s, Ollama's 179 s and KTransformers' 946 s. OpenClaw's watchdog fires at 120 s, so this is availability, not latency.
Host RAM holds the whole pool, 140 GB for DSV4-Flash. Linux, NVIDIA, CUDA 13 only. Budget DRAM before VRAM.
Explain It Like I'm New
Modern open AI models are enormous, often hundreds of billions of parameters, but the newest ones have a useful property. They are built from hundreds of small specialist sub-networks called experts, and any single word the model generates only wakes up a handful of them. The computation is small. The storage is not, because you never know in advance which handful you will need.
That gap is why running these models at home has been frustrating. Your graphics card has fast memory but not much of it. Your regular system RAM is roomy but slow, and the cable connecting the two is a bottleneck. Every time the model needs an expert that is not already on the graphics card, something has to give.
Every previous local AI tool answered this the same way: get better at guessing which experts you will need next, and copy them over early. Sensible, and it has a ceiling. No matter how good the guessing gets, a wrong guess still means waiting on that cable.
FreeToken asks a different question. If an expert is already sitting in system RAM, why move it at all? Your CPU is right there. So it measures two speeds on your specific machine, how fast the cable copies and how fast the CPU computes, and splits the work between them in whatever ratio your hardware happens to favor. A cheap laptop with a narrow cable sends most of the work to the CPU. A high-end desktop with a fat cable sends most of it across.
The consequence is that a laptop becomes a serious inference machine, not a compromised one. And it reframes what local AI is limited by. Not the graphics card, but how well the software orchestrates the whole machine.
See It In Action
Independent video coverage is thin, the release is under a week old. What exists and is worth your time:
FreeToken desktop app (flashml.ai), the one-click Windows and Linux build with a GUI for model selection, chat and engine tuning. Fastest way to see the engine's behavior without touching CUDA toolchains. The repo carries a console screenshot if you want a preview first.
Quickstart and CLI reference (docs/quickstart.md, docs/cli.md), the highest signal material in the project. Every flag in the MoE offload table maps directly onto a mechanism in the paper. Read
ft bench bwandft ctl cachefirst.Agent harness integration (
ft launch claude), points Claude Code, Codex, OpenCode, OpenClaw or the DeepSeek harness at your local server and strips cloud API keys from the child environment. This is the demo that actually communicates the thesis.KTransformers, SOSP 2025 (repo), the strongest baseline here and the system that made AMX-optimized in-place CPU expert execution fast. Understand it and FreeToken's contribution sharpens into focus.
Developer Slack and Community Discord, both linked from the repo header, where the hardware-specific tuning questions are being answered right now.
Community Conversation
FlashML-org/FreeToken on GitHub, Apache 2.0, trending within days of release, roughly 108 stars at the six day mark and climbing. Adoption is early, so treat the reported benchmarks as the authors' numbers until someone reproduces them.
Trendshift repository tracker, captures the reaction that matters most. The recurring reply to the 284B demo is a variant of "I was hoping this was on a single 5090," from readers who then found the 192 GB system RAM line. The VRAM story travels faster than the DRAM story, and the DRAM story is the real constraint.
Hugging Face paper page, the discussion thread for the paper, and the fastest place to see whether the systems community accepts the
q*derivation.Chinese developer channels have driven a large share of early attention, framed around not needing a multi-GPU workstation. The pitch that landed there was capability unlock, an 8 GB laptop running a 35B MoE, rather than throughput.
Reproduction trackers currently mark the paper as not independently verified with no benchmark numbers confirmed. Worth stating plainly given how strong the claims are.
Local Inference Is A Scheduling Problem Now, Not A Capacity One
The useful thing FreeToken proves is not that a laptop can run a 35B model. It is that the question "does this model fit on my GPU" has stopped being the right question. Sparse activation already made the computation feasible. What was left was orchestration, and orchestration was being handled by static placement decisions frozen at load time, which we can now measure performing at chance.
Two scalars, measured on your own machine, beat three years of expert prediction research. That should be uncomfortable, and it should also be instructive about where the remaining wins in local inference actually are.
References
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, the paper, arXiv 2608.16157
FlashML-org/FreeToken, Apache 2.0 implementation and docs
KTransformers, SOSP 2025, the strongest hybrid CPU/GPU baseline
MoE-Infinity: Efficient MoE Inference on Personal Machines, sparsity-aware expert cache with activation tracing
Fiddler: CPU-GPU Orchestration for MoE Inference, first treated a missed expert as work, not just data
Local Routing Consistency of MoE Language Models, the measurement the LRU cache rests on
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management, per-step schedule simulation, the approach graph capture rules out
WiSP: A Working-Set View of MoE Serving, independently finds single-stream decode PCIe bound regardless of prediction accuracy
TraceLab: Characterizing Coding Agent Workloads for LLM Serving, source of the 33 tok/s Codex production median
SGLang: Efficient Execution of Structured Language Model Programs, the radix prefix tree and serving substrate
FreeToken is an edge-native MoE serving engine that keeps the full expert pool in host RAM and splits each decode step's cache misses between PCIe transfer and in-place CPU execution using a closed-form ratio of two bandwidths measured on the deployed machine. The architectural insight is that the split had to be trivial to compute, because anything heavier cannot survive CUDA Graph capture, and that constraint is what competing schedulers keep violating. It matters because it moves the limit on local inference from GPU capacity to software orchestration, putting a 35B model on an 8 GB laptop at 39.3 tok/s and a 753B model on a single workstation card.
Keep Going
If this was useful, the thing to take with you is the method, not the engine. Read a serving system by asking what it measures on your machine and what it froze at load time. That one question separated FreeToken from every baseline in this issue.
SnackOnAI runs this teardown weekly on the systems engineers actually deploy, serving engines, kernels, caches and the benchmarks vendors leave out. No announcements, no press release summaries. Subscribe at snackonai.com and join 10,000+ engineers reading it.
Forward this to whoever on your team is about to spec a local inference box by VRAM.
Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad, it helps us keep building and delivering value 🚀
Build. Break. Fix. Learn.
KodeKloud gives you 1,280+ hands-on labs where you provision Kubernetes clusters, write Terraform configs, build CI/CD pipelines, configure Linux systems, containerize apps with Docker, automate with Ansible, and manage Git workflows.
78+ playgrounds let you experiment freely in sandbox AWS environments, Kubernetes clusters, and CI/CD systems without risk.
190+ courses across DevOps, Cloud, and AI pair theory with hands-on labs at every step.
KodeKloud Engineer and 100 Day Challenges provide real-world job scenarios with automated grading that confirms your solutions work.
Stuck? The 55,000+ member Discord community connects you with peers and instructors ready to help.
Every lab runs in a live environment. You deploy, you troubleshoot, you learn. No videos without context. No simulations. The kind of practice that actually builds confidence because you've done real work, not watched someone else do it.