In partnership with
That reframing produced a 525% throughput gain in simulation and a 75% request capacity increase under real Kimi workloads. Kimi K3, the 2.8T-parameter model now running on top of it, is what happens when you scale that infrastructure to the frontier and redesign attention to serve it.
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | July 20, 2026
What It Actually Does
Kimi K3 (kimi.com/blog/kimi-k3) is Moonshot AI's flagship model, released July 16, 2026: 2.8 trillion total parameters, 16 of 896 experts active per token (roughly 50B active parameters per forward pass), a 1M-token context window, and always-on thinking mode. It is the first open-source model to reach the 3-trillion-parameter class.
Underneath it runs Mooncake (arXiv:2407.00079), the production inference platform that disaggregates prefill and decoding into separate GPU clusters and implements a distributed KVCache across CPU DRAM and SSD in addition to GPU HBM.
Two architectural innovations define K3's design: Kimi Delta Attention (KDA), a hybrid linear attention mechanism that delivers 6.3x faster decoding at 1M context versus standard attention, and Attention Residuals (AttnRes), which selectively retrieves representations across depth at roughly 2% compute cost but 25% training efficiency improvement. Below those sits Stable LatentMoE with Quantile Balancing, which replaced the fragile auxiliary-loss-based load balancing that causes MoE instability at scale.
Scope covered: Mooncake's disaggregated architecture and KVCache scheduler, K3's new architectural primitives (KDA, AttnRes, Stable LatentMoE), API behavior, quantization-aware training strategy, and benchmark positioning. Excluded: the Kimi-Audio architecture (covered separately in arXiv:2504.18425), fine-tuning internals, and the forthcoming K3 technical report details.
The Architecture, Unpacked
Mooncake: The Serving Platform
Conventional LLM serving collocates prefill and decoding on the same GPU. Prefill, the compute-heavy phase that processes the input prompt, competes with decoding, the memory-bandwidth-heavy phase that generates tokens one at a time. They have fundamentally different resource profiles, so co-locating them means neither gets its optimal resource allocation.
Mooncake's answer is disaggregation:
Caption: The critical path is not GPU compute: it is KVCache transfer from prefill to decoding cluster. The scheduler's job is to maximize reuse across all three storage tiers.
The key insight in the Mooncake paper is that prefill nodes produce KVCache as output and decoding nodes consume it as input. If you treat KVCache transfer as the primary constraint to optimize, rather than GPU utilization, you expose a completely different set of optimizations: prefix caching across requests, tiered storage for cold KVCaches, and a scheduler that rejects requests early when SLOs cannot be met, rather than queuing them and degrading latency for all pending requests.
The results are concrete: 525% throughput improvement in simulated long-context scenarios, 75% more requests handled under real Kimi workloads, and a claimed cache hit rate above 90% in production coding workloads.
Kimi K3: The Model Architecture
K3 introduces three novel architectural components on top of an MoE transformer backbone:
Kimi Delta Attention (KDA): A hybrid linear attention mechanism. Standard attention scales quadratically with sequence length (O(n²)). Linear attention approximates the attention operation in O(n) by decomposing the softmax operation. KDA combines linear attention with a residual full-attention component, capturing long-range dependencies efficiently while maintaining the expressivity needed for complex reasoning. At 1M context, this produces 6.3x faster decoding than standard attention.
Attention Residuals (AttnRes): Rather than accumulating representations uniformly across all transformer layers, AttnRes selectively retrieves representations from earlier layers based on relevance. Think of it as a learned skip connection that allows the model to "reach back" to earlier processed information without forcing every layer to carry it forward. Cost: 2% additional compute. Benefit: 25% improvement in training efficiency (more capability per FLOPs invested).
Stable LatentMoE with Quantile Balancing: Standard MoE uses an auxiliary loss to encourage expert load balance. The auxiliary loss coefficient is a sensitive hyperparameter that breaks at scale: too low, experts become imbalanced; too high, routing degrades into uniform distribution, destroying specialization. Quantile Balancing replaces the auxiliary loss by deriving expert allocation directly from the distribution of router scores. When the top-k router scores fall above a quantile threshold, routing proceeds normally. When imbalance is detected via score distribution, the allocation is rebalanced without touching the loss function. No heuristic hyperparameter on the critical path.
Caption: Each K3 block applies KDA first, then AttnRes selective retrieval, then MoE. The data flow from previous blocks into AttnRes is what allows depth-efficient scaling.
The Code, Annotated
Snippet One: Kimi K3 API with Streaming Thinking
import os
from openai import OpenAI
# ─── Client setup: OpenAI-compatible, different base_url ─────────────
# K3 uses the Moonshot API endpoint but accepts the same SDK interface.
# This is a deliberate compatibility choice: no new SDK, no migration.
client = OpenAI(
api_key=os.environ["MOONSHOT_API_KEY"],
base_url="https://api.moonshot.ai/v1",
)
# ─── Streaming with separate reasoning and answer deltas ─────────────
stream = client.chat.completions.create(
model="kimi-k3",
messages=[{"role": "user", "content": "Explain why prefix caching matters for serving large models."}],
stream=True,
# reasoning_effort="max" is the default and currently the only option.
# K3 always has thinking enabled — you cannot turn it off.
# This is different from K2.x which had a separate "thinking" param.
# ← THIS is the design choice: thinking is always-on, not opt-in.
)
reasoning_tokens = 0
answer_tokens = 0
for chunk in stream:
delta = chunk.choices[0].delta
# ← reasoning_content streams the thinking trace
# ← content streams the final answer
# They are separate fields, not interleaved.
# This separation matters: you can count and bill them differently.
reasoning = getattr(delta, "reasoning_content", None)
if reasoning:
print(reasoning, end="", flush=True)
reasoning_tokens += len(reasoning.split())
if delta.content:
print(delta.content, end="", flush=True)
answer_tokens += len(delta.content.split())
print(f"\n[thinking ~{reasoning_tokens} tokens, answer ~{answer_tokens} tokens]")
Caption: The reasoning_content / content split is not cosmetic: it reflects the underlying training separation between chain-of-thought tokens and output tokens, and allows the serving layer to bill them at different rates.
Snippet Two: Context Caching with the 1M Window
from pathlib import Path
# ─── Load a large knowledge base (e.g. an entire codebase as text) ────
# This is the primary use case for the 1M context window.
# The key design: context caching is AUTOMATIC. No cache IDs, no TTLs.
# If you keep the prefix unchanged, Mooncake's KVCache layer handles it.
knowledge: str = Path("codebase_context.md").read_text(encoding="utf-8")
# Assume ~500K tokens. At cache-miss pricing: $3.00/MTok input.
# ← THIS is why prefix caching matters: repeated queries over the
# same long prefix pay $0.30/MTok (cache-hit rate) instead.
# At >90% cache hit rate in coding workloads, the effective input
# price is ~$0.30 per MTok, not $3.00. That is 10x cheaper.
questions = [
"What database migrations have run in the last 30 days?",
"Find all places where we override the default retry policy.",
"Which services depend on the legacy auth module?",
]
for question in questions:
completion = client.chat.completions.create(
model="kimi-k3",
messages=[
{"role": "system", "content": knowledge},
# ← Keep this prefix IDENTICAL across calls.
# Any change to the prefix (even a space) invalidates the cache.
# Mooncake's prefix tree matches longest common prefix.
{"role": "user", "content": question},
],
)
print(f"Q: {question}")
print(f"A: {completion.choices[0].message.content}\n")
# First call: cache miss → $3.00/MTok for the 500K-token prefix
# Subsequent calls: cache hit → $0.30/MTok
# Savings on 3 questions with 500K prefix: ~$1.35 saved per round
Caption: The pricing structure ($3.00 cache miss, $0.30 cache hit) makes automatic prefix caching not a convenience feature but a fundamental cost architecture. Mooncake's KVCache storage hierarchy is what makes >90% hit rates achievable in production.
It In Action: End-to-End Worked Example
Scenario: Codebase-scale reasoning with a 1M context window.
Input: A 350,000-token context containing an entire mid-size Python monorepo, system prompt, and a multi-step debugging request.
Step 1: First call (cache miss)
Input tokens: 350,000
Cache status: MISS (first call, no KVCache stored)
Prefill cost: 350,000 × $3.00/MTok = $1.05
Output tokens: ~800 (analysis + corrected function)
Output cost: 800 × $15.00/MTok = $0.012
Total cost: ~$1.062
Latency (prefill): ~18s (compute-heavy, Mooncake prefill cluster)
Latency (first token): ~18.4s
Step 2: Follow-up call on same context (cache hit)
Input tokens: 350,000
Cache status: HIT (Mooncake matched prefix, served KVCache from storage)
Prefill cost: 350,000 × $0.30/MTok = $0.105
Output tokens: ~400
Output cost: 400 × $15.00/MTok = $0.006
Total cost: ~$0.111
Latency (prefill): ~2.1s (KVCache loaded, no recomputation)
Latency (first token): ~2.5s
Step 3: K3's KDA effect at 350K context Standard attention (K2.x): prefill memory scales as O(n²), making 350K tokens prohibitively expensive. KDA hybrid linear attention: memory scales as O(n), allowing 350K-token prefill to complete in the same compute budget as a ~100K standard attention prefill.
Real numbers from production:
Cache hit rate in coding workloads: >90% (Moonshot stated)
Mooncake throughput improvement vs. baseline: 525% (simulated), 75% (real production)
K3 Frontend Code Arena rank: #1 at 1,679 points (LMArena, July 16, 2026)
K3 DeepSWE score: 67.3 (KimiCode harness)
K3 BrowseComp: 90.4 (1M context, no context management)
Scaling efficiency improvement over K2: ~2.5x
Why This Design Works (and What It Trades Away)
The Mooncake architecture's core bet is that KVCache reuse is the dominant optimization opportunity in production LLM serving, especially for long-context workloads. This is correct for Kimi's workload: users asking multiple questions over the same document, coding sessions with persistent context, and agentic workflows that revisit the same codebase repeatedly. In those scenarios, recomputing the prefill from scratch on every request is pure waste.
The tradeoff is infrastructure complexity. Disaggregated prefill and decoding clusters require high-bandwidth interconnects for KVCache transfer. The KVCache must be moved from the prefill GPU to a storage tier and then to the decoding GPU, adding latency on the KVCache transfer path. For short prompts with no prefix overlap, the disaggregated architecture adds overhead with no benefit.
KDA solves a genuine problem but it introduces a new one: prefix caching for hybrid linear attention is not compatible with standard vLLM PagedAttention. Moonshot contributed a KDA-aware prefix caching implementation to the vLLM community (to be released with the weights on July 27). If that contribution does not land cleanly, self-hosters will face degraded cache hit rates at inference time, significantly increasing their effective token cost.
The Stable LatentMoE decision to remove the auxiliary loss balancing hyperparameter is the right call at 896 experts. Every MoE model that tries to scale past roughly 64 experts with auxiliary loss balancing hits the same pathology: the coefficient that worked at 64 experts does not generalize to 896, requiring expensive re-tuning runs. Quantile Balancing, which derives allocation from the empirical distribution of router scores, is hyperparameter-free by construction.
Quantization-aware training from the SFT stage onward using MXFP4 weights with MXFP8 activations is not a post-hoc compression. It is a design statement: K3 is built to be deployed at MXFP4 precision, targeting hardware with MXFP4 tensor core support (NVIDIA Blackwell, AMD MI300X in MXFP4 mode). This is the reason Moonshot recommends deploying K3 on supernode configurations with 64 or more accelerators: KDA's linear attention produces larger high-bandwidth communication domains that benefit from larger collective groups.
Technical Moats
The Mooncake KVCache tiered storage implementation is the hardest piece to replicate. Managing a distributed KVCache across GPU HBM, CPU DRAM, and SSD with a prefix-matching scheduler, transfer queuing, and eviction policy is not a weekend project. The paper describes the architecture; the engineering of a production system that achieves >90% cache hit rates under real traffic is years of infrastructure work.
Quantile Balancing for MoE at 896 experts is the most novel contribution from a research perspective. MoE balancing at this expert count without an auxiliary loss requires either careful curriculum design or a stable hyperparameter-free routing strategy. Quantile Balancing is the latter. Reproducing it requires understanding the distribution of router scores at different training stages, which requires the training data and intermediate checkpoints.
The fully balanced expert-parallel training method with static shapes and no host synchronization is an inference throughput enabler. At 896 experts across large EP (expert parallel) groups, the naive approach requires host-side synchronization to handle imbalanced expert load, which stalls the GPU. Static shape allocation with offline-profiled load estimates removes this bottleneck. This is a systems optimization that requires tight co-design between the training team and the inference infrastructure team.
Contrarian Insights
Insight One: The "2.8T parameter" headline is actively misleading for self-hosting capacity planning. The number that matters for hardware is active parameters per forward pass: roughly 50B (16 of 896 experts). At 50B active parameters with MXFP4 weights, per-token compute is more similar to a 50B dense model than a 2.8T model. But the weight storage is 2.8T parameters at MXFP4 precision: approximately 350GB just for weights. That number does not fit on a single node with standard GPU configurations. Self-hosting K3 at full precision requires 16-18 accelerators in a tightly coupled supernode. At 1-bit quantization (community builds post-weight-release), 4-6 accelerators. The "open source" announcement is real, but the accessible open source will be 4-bit or lower community quantizations, not the official MXFP4 weights.
Insight Two: K3's hallucination rate rose alongside its accuracy gains, and the benchmark suite does not adequately capture this tradeoff. The LMArena community analysis notes a higher hallucination rate in K3 versus K2.6, despite K3's strong benchmark performance. This is a classic recall-precision tradeoff at the model level: training for aggressive long-horizon agentic behavior and high benchmark scores on coding tasks produces a model that improvises more when uncertain. K3's documentation explicitly lists "excessive proactiveness" as a known limitation: it may "make unexpected decisions on the user's behalf" when encountering ambiguous intent. For production applications requiring conservative behavior, that is not a benchmark number. It is a deployment risk that requires explicit behavioral constraints in the system prompt.
Surprising Takeaway
K3 autonomously optimized the GPU kernels used in its own architecture. In one of the case studies in the technical blog, an early version of K3 handled "the majority of the team's kernel optimization work" during late-stage development, including KDA and AttnRes kernels on NVIDIA H200. The model also built MiniTriton, a Triton-like compiler with its own IR layer, optimization passes, and PTX code generation pipeline, achieving performance "on par with or better than Triton" on supported benchmarks. A model that can write the GPU kernels that run itself, and does so in production before the model is released, is not a capability demo. It is a signal that the capability-compute flywheel is accelerating: better models write better infrastructure, which enables better models.
TL;DR For Engineers
Kimi K3 (released July 16, 2026): 2.8T total / ~50B active parameters, 896 experts with 16 active per token, 1M-token context, native vision, always-on thinking via
reasoning_effort. Weights releasing July 27, 2026. API live now atapi.moonshot.ai/v1with OpenAI-compatible SDK.Two new architectural primitives: KDA (hybrid linear attention, 6.3x faster at 1M context) and AttnRes (selective depth retrieval, 25% training efficiency at 2% compute cost). Together they produce a ~2.5x scaling efficiency improvement over K2.
Mooncake serving platform: disaggregated prefill/decoding clusters with KVCache tiered storage (GPU HBM → CPU DRAM → SSD). Achieves >90% cache hit rate in coding workloads. Cache-hit pricing is $0.30/MTok vs. $3.00/MTok cache-miss for 10x effective input cost reduction.
Stable LatentMoE with Quantile Balancing removes the auxiliary loss hyperparameter from MoE routing, enabling stable training at 896-expert scale. Quantization-aware training starts at SFT using MXFP4 weights, MXFP8 activations.
Frontend Code Arena #1 at 1,679 points (LMArena). BrowseComp: 90.4 (1M context). DeepSWE: 67.3. Trails Claude Fable 5 and GPT 5.6 Sol on overall intelligence benchmarks. Hallucination rate higher than K2.6.
Explain It Like I'm New
When you ask a large language model a long question, it first processes your entire input — that processing is called "prefill." Then it generates the answer one word at a time. These two phases need different kinds of computing resources: prefill needs raw computation, generation needs fast memory access. Most existing systems run both phases on the same hardware, so they fight for resources.
Mooncake separates them onto different machines and introduces a third innovation: a shared memory system for "KV cache," which is the model's intermediate working memory during processing. If someone asks similar questions over the same document repeatedly, instead of reprocessing the entire document each time, Mooncake stores that intermediate work and reuses it. This is roughly like a professor who, after reading a 500-page textbook once, can answer student questions about it immediately, without re-reading.
Kimi K3, the model running on Mooncake, introduces new attention mechanisms designed for very long documents. Standard attention gets quadratically more expensive as context grows: double the document, quadruple the cost. K3's hybrid attention is roughly linear: it trades some precision for dramatically lower memory cost at scale.
The combination of a purpose-built inference infrastructure and a new attention architecture is what makes a 1-million-token context window practical, not just theoretically possible.
See It In Action
Kimi K3 Technical Blog Source: Moonshot AI | https://www.kimi.com/blog/kimi-k3 The official blog contains the MiniTriton GPU compiler case study, the chip design demo, and the gravitational-wave analysis workflow, all of which demonstrate K3's long-horizon agentic capabilities concretely rather than through benchmarks.
Mooncake Paper (arXiv:2407.00079) Source: Ruoyu Qin et al., Moonshot AI | https://arxiv.org/abs/2407.00079 The full paper contains the disaggregated architecture diagrams, scheduler design, SLO formulation, and the experiments showing 525% throughput improvement. Essential reading for engineers building on top of Mooncake's serving model.
Kimi K3 Playground Source: Kimi API Platform | https://platform.kimi.ai/playground Direct access to K3 Max with the 1M context window before the weights release. The reasoning trace is streamed separately, making the thinking process directly inspectable.
FrontierSWE Leaderboard Source: frontierswe.com | https://www.frontierswe.com The benchmark where K3 is evaluated, with independent harness comparisons. The evaluation notes for K3 contain important caveats about harness compatibility (KimiCode vs. Claude Code vs. Codex) that affect score interpretation.
Community Conversation
Kimi (@Kimi_Moonshot) on X https://x.com/Kimi_Moonshot The official announcement thread covers the Frontend Code Arena result (1,679 points, #1, 17-place jump over K2.6) and the chip design case study. Moonshot explicitly states K3 trails Claude Fable 5 and GPT 5.6 Sol overall, which is an unusual level of honesty in a model launch post.
Tom's Hardware: "Kimi K3 Beats Claude Fable 5 in Frontend Code Arena" https://www.tomshardware.com/tech-industry/artificial-intelligence/moonshot-releases-2-8-trillion-parameter-kimi-k3 The article includes the Bank of America analyst note calling K3 evidence that "large-scale pre-training plus architectural work can still deliver step-change gains despite compute constraints" — a significant framing given US export controls on high-end GPU access to China.
Awesome Agents analysis: "Kimi K3 — price and hallucination tradeoffs" https://awesomeagents.ai/models/kimi-k3/ The most balanced third-party breakdown: K3 tops Frontend Code Arena but ranks only #9 in general Text Arena. Pricing tripled versus K2.6. Hallucination rate increased. Useful counterweight to the benchmark-only framing in most coverage.
Hugging Face: "Kimi K3 Model Overview: MXFP4 Quantization and Open Weights" https://huggingface.co/blog/ResterChed/kimi-k3-model-overview-mxfp4-quantization-open-wei Technical analysis of the quantization-aware training approach and what the July 27 weight release means practically for the self-hosting community.
Open Weights Do Not Mean Accessible Weights
Kimi K3 is open. The weights release on July 27. The community will quantize, distill, and adapt it within days, exactly as happened with Llama, Qwen, and DeepSeek. That is the correct prediction.
What is less clear is whether the Mooncake serving innovations, specifically the KVCache tiered storage and disaggregated prefill/decoding architecture, will be available to self-hosters in a form that reproduces the >90% cache hit rates. The KDA-aware vLLM contribution is a promising signal. But a production inference system is not a model checkpoint. The gap between "weights available" and "production-grade serving infrastructure available" is where the real moat lives.
K3 is the first open 3T-class model. The question worth asking is not whether it beats Claude Fable 5 on every benchmark (it does not). It is whether "open source at the frontier" means the same thing in 2026 as it did in 2023. When the model itself can write the GPU kernels for its own architecture, the answer is probably not.
References
Kimi K3 is Moonshot AI's 2.8T-parameter open-source flagship, built on Kimi Delta Attention (6.3x faster at 1M context), Attention Residuals (25% training efficiency at 2% cost), and Stable LatentMoE with Quantile Balancing, served by the Mooncake KVCache-centric disaggregated inference platform that achieves >90% cache hit rates and 525% throughput improvement in long-context scenarios. It is the first open 3T-class model and sits directly below Claude Fable 5 and GPT 5.6 Sol on overall benchmarks, while leading the Frontend Code Arena at 1,679 points. The real story is not the parameter count but the co-design between the model architecture and the serving infrastructure that makes 1M-context deployment economically viable.
Sponsored Ad
If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad — it helps us keep building and delivering value 🚀
AI help, without the trust tax.
Most AI tools ask you to trade your data for intelligence. Norton Neo doesn't. It's the first safe AI-native browser built by Norton, and it gives you powerful built-in AI without handing your privacy over to get it. Search, summarize, and write with AI built directly into your browser. Your data stays yours. Your context stays private.
Built-in VPN, anti-fingerprinting, and ad blocking come standard. No add-ons. No setup. No compromises.
Fast. Safe. Intelligent. That's Neo.