DeepSeek's DeepInfer Shows the Engine Was Never the Hard Part

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 21, 2026

The promise: by the end of this issue you will understand what DeepInfer actually open sources, how DeepSeek serves 155 trillion tokens a day around it, and what you would still have to build to run it in production.

The fence: we cover the DeepInfer repository, its deployment Q&A, DeepSeek-V4.1-Flash as the served model, and the background papers on DeepSeek-V3 and PagedAttention. We skip model training, the deepseek-recipe tokenizer internals beyond what matters for serving, and benchmark claims about model quality.

What It Actually Does

DeepInfer is the inference framework behind DeepSeek's production API, released under Apache 2.0. The open release supports one model, DeepSeek-V4.1-Flash, on two platforms: NVIDIA SM100 (Blackwell) and Huawei Ascend 950. It began as a fork of vLLM, with ideas borrowed from SGLang and TensorRT-LLM.

What ships:

Feature	What it means
Kernels	FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, custom TileLang ops
Parallelism	Attention data parallelism, MoE expert parallelism (EP) with load balancing (EPLB)
DSpark	Speculative decoding with confidence scheduled, variable length verification
SWA bounded replay	Recompute only a short suffix instead of the whole prompt
Async scheduling	CPU schedules step N+1 while the GPU runs step N
Batch invariance, deterministic sampling	Identical outputs regardless of batch mix (NVIDIA only)
Structured outputs	Regex, JSON schema, or EBNF constrained decoding

What does not ship: the KV cache system, encoder/prefill/decode disaggregation, and load balancing across data parallel replicas. DeepSeek says those parts are "tightly coupled to our internal infrastructure."

The model it serves, per its card: 552B total parameters, 8B active in prefill and 16B in decode, 384 routed experts with 6 active, a 1M token context, MIT licensed, and a KV cache of 890 bytes per token.

The honest one line summary: a production grade engine core, released without the distributed systems that make it cheap at scale.

The Architecture, Unpacked

The repository is the engine. The Q&A describes the system around it, and that system is where most of the design lives.

Caption: Focus on the labels marked internal. The open engine sits in the middle; the components that decide cost per token, cache hits and load balance, wrap around it.

Four decisions define the system, ranked.

One: wide expert parallelism for decode. Decode runs at EP64 to EP128: the MoE experts are spread across 64 to 128 GPUs in one supernode, so each GPU holds few experts and serves huge batches of them. That cuts per step memory traffic for expert weights. Prefill does the opposite, using the smallest EP that fits in memory.

Two: a KV cache measured in petabytes, not gigabytes. Block KV lives on a dedicated 3FS cluster rather than local host memory, so any prefill machine can hit any prefix. That removes the classic conflict between prefix affinity routing and load balancing. Compute nodes fail more than an order of magnitude more often than storage nodes, so cache survives node loss.

Three: make recomputation constant. SWA (sliding window attention) layers need a fresh window after a cache hit. Bounded replay recomputes only the last 128 tokens, so recovery cost is independent of context length. In the decoder layers, this nearly halves total prefill compute.

Four: verify only the draft tokens worth verifying. DSpark drafts 5 tokens, predicts each one's acceptance odds with a confidence head, and verifies only the prefix with the highest expected payoff.

The Code, Annotated

First, the real entry points, from the repository's README and examples.

# Linux, GCC ≥ 13, Python 3.12, CUDA 13 (or CANN 9.2 for Ascend)
export DEEPINFER_ACCELERATOR=nvidia          # or ascend; keep it exported at runtime too
./build.sh                                   # builds vendored DeepGEMM, DeepEP, FlashMLA

# Speculative decoding with confidence-scheduled, variable-length verification
export DEEPINFER_ENABLE_MTP_VARLEN=1
export DEEPINFER_USE_NEXTN_DECODE=5          # max draft length (default 5)

# Decoder SWA bounded replay with the default V4.1 settings
export DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
export DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140   # ← THIS is the trick: 128 window + speculative slack

python examples/offline_inference.py --ep 8  # downloads DeepSeek-V4.1-Flash-1017 on first run

import json
from deepinfer.sampling_params import SamplingParams

json_schema = {
    "type": "object",
    "properties": {"color": {"type": "string", "enum": ["red", "green", "blue"]},
                   "count": {"type": "integer", "enum": [1, 2, 3]}},
    "required": ["color", "count"], "additionalProperties": False,
}
# The repo's own test deliberately asks for a long story while forcing JSON:
# constrained decoding must win over the prompt. Grammars compile off the hot path.
params = SamplingParams(temperature=0, max_tokens=128,
                        so_args={"schema_type": "json", "schema": json.dumps(json_schema)})

Caption: Every major feature is an environment variable or a sampling argument. The KEEP_TOKENS=140 value must cover the 128 token sliding window plus speculative backtracking, or cache hits produce wrong attention states.

Second, how the prefill scheduler sizes a chunk. The Q&A says each GPU solves a fitted quadratic cost model for the largest chunk that fits a time budget. Here is that logic, runnable; the coefficients are ours, for illustration.

import math

MAX_TOKENS = 32_768

def cost_ns(q, c, a=2.0e-3, b=2.0e-2, d=9.0e3):
    """Step cost for q new tokens on top of a cached prefix of c tokens:
    a*q^2 attention inside the chunk + b*q*c attention to the prefix + d*q linear work.
    DeepSeek fits these coefficients per deployment; ours are illustrative."""
    return a*q*q + b*q*c + d*q

def solve_chunk(budget_ns, c, a=2.0e-3, b=2.0e-2, d=9.0e3):
    B = b*c + d
    q = (-B + math.sqrt(B*B + 4*a*budget_ns)) / (2*a)   # ← THIS is the trick: solve, don't guess
    return min(MAX_TOKENS, int(q))

budget = 400e6   # the DP group agrees on a budget via all-reduce MIN, so no rank straggles
for prefix in (0, 64_000, 256_000, 900_000):
    q = solve_chunk(budget, prefix)
    print(f"cached prefix {prefix:>7,} → chunk {q:>6,} tokens, {cost_ns(q, prefix)/1e6:5.1f} ms")
# cached prefix       0 → chunk 32,768 tokens, 297.1 ms
# cached prefix  64,000 → chunk 32,768 tokens, 339.0 ms
# cached prefix 256,000 → chunk 28,215 tokens, 400.0 ms
# cached prefix 900,000 → chunk 14,798 tokens, 400.0 ms

Caption: Fixed chunk sizes make long context requests blow up step latency. Solving for the chunk keeps every step near the same time budget, so a 900K token prompt does not stall everyone else in the batch.

Third, the cache hit policy, reimplemented from the Q&A's rules.

BLOCK, BACKTRACK = 128, 6
CAP = BLOCK + BACKTRACK            # 134: never reuse the final 134 tokens of an input

def hit_length(L, block_hit, swa_has):
    limit = (L - CAP) // BLOCK * BLOCK
    L1 = min(block_hit, limit)      # block KV decides HOW MUCH can be reused
    if L1 <= 0:
        return 0
    return L1 if swa_has(L1) else L1 - BLOCK   # SWA KV decides whether to back off one block

print(hit_length(50_000, 49_920, lambda p: True))    # 49792: full reuse
print(hit_length(50_000, 49_920, lambda p: False))   # 49664: replay 128 tokens instead

Caption: Two caches, two jobs. The big paged block cache sets the hit length; the tiny 1.4 MB SWA snapshot only decides whether to recompute one 128 token block. A missing snapshot costs a constant, not a full prefill.

It In Action

Input: an agent sends a request with a 131,072 token context to DeepSeek-V4.1-Flash, then sends a follow up two hours later with the same prefix plus a short new message.

Step one, entry. deepseek-recipe encodes the request in parallel segments split at known special token positions. The Entrypoint runs a hash chain prefix match against KV metadata before routing.

Step two, KV size. At 890 bytes per token, the full context's KV cache is 116.7 MB. For comparison, DeepSeek-V3's MLA cache works out to about 70 KB per token by our math, 79x larger.

Step three, prefill to decode. Moving 116.7 MB across the recommended 400 Gbps per 8 GPUs takes about 2.3 ms. That is why DeepSeek can split prefill and decode across adjacent data centers.

Step four, decode. On EP64/DP64 with 3 draft tokens at 75% acceptance, DeepSeek's chart shows above 25,000 tokens/s per GPU at 256 requests per rank, near 100 tokens/s per user.

Step five, the follow up. Two hours later, well inside the 3 day agent TTL, the block KV is still on 3FS. If the 1.4 MB SWA snapshot is also present, everything but the last 134 tokens is reused; if not, 128 extra tokens are replayed.

Output: the follow up pays for prefill on a few hundred tokens instead of 131,072.

Decode concurrency (seqs/rank)	Fixed 5 verify	DSpark dynamic	Gain	Per user tok/s (dynamic)
16	4,891 tok/s	4,816	−1.5%	301
60 (production)	13,730	14,466	+5.4%	241
96	16,817	18,660	+11.0%	194

Caption: DSpark is a throughput feature. At low concurrency it is 1.5% slower; it pays off as batches grow, which is exactly when per user speed is already falling.

Why This Design Works, and What It Trades Away

Why it works. Decode cost is dominated by moving expert weights and KV through memory. Wide EP shrinks the first and an 890 byte per token KV shrinks the second. Prefill cost is dominated by recomputation, which a 72 hour, petabyte scale cache plus constant cost replay largely eliminates. The scheduler and async pipeline remove CPU induced GPU idle time. At 155 trillion tokens per day per cluster, about 1.79 billion tokens per second, DeepSeek reports revenue of $38 per GPU hour at off peak pricing.

What it trades away.

Minimum scale. The smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs.
Hardware reach. SM100 only on NVIDIA. No Hopper, which is most of today's installed base.
Platform parity. Batch invariance and deterministic sampling are NVIDIA only, and all published benchmarks are NVIDIA.
Model scope. One model. vLLM covers hundreds of architectures.
Completeness. Offline inference only; no online server, no disaggregation, no KV store in the release.

Technical Moats

The moat is the storage and balancing layer DeepSeek did not release. The Q&A says the load balancers reach about 80% of a perfectly balanced offline benchmark. A self hosted deployment without them starts below that. The KV store requires about 14.4 TB of logical cache per GPU: for the minimum 128 GPU unit, that is 1.84 PB logical, at least 4 storage servers of 20 × 30.73 TB SSDs after 10+2 erasure coding by our math, before any headroom.

The second moat is co-design of model and engine. The 890 byte KV, the 128 token SWA window, the encoder decoder split that lets decoder layers project KV from one hidden state, and the replay rules were designed together. A general engine like vLLM can adopt the kernels, but not the assumptions baked into the model.

Contrarian Insights

Insight One: "open source inference engine" undersells what is missing. DeepSeek released the parts that make one machine fast and kept the parts that make a fleet cheap: the KV cache system, disaggregation, and replica load balancing. Their own architecture recommends retaining KV for 72 hours across roughly 100 PiB of SSD, and their TTL tuning shows cache hit rates move revenue (a 3 day TTL costs chatbot traffic 0.5% to 1% of hit rate). Teams expecting to clone DeepSeek's API economics from this repository will get the engine, then discover the economics live in the infrastructure around it.

Insight Two: Huawei got day one support, not day one parity. Ascend 950 is a first class build target, which matters geopolitically and for China's buildout of Ascend capacity. But batch invariant inference and deterministic sampling are NVIDIA only, and every published throughput chart was measured on SM100 GPUs. Meanwhile NVIDIA support skips Hopper entirely. In practice the release targets a narrow slice of hardware on both sides: Blackwell or Ascend 950, in supernodes of 64 or more.

Surprising Takeaway

At 890 bytes per token, a full 131K token context fits in 117 MB and crosses a 400 Gbps link in about 2.3 milliseconds. KV cache has become so small that DeepSeek treats it as durable data, keeping it for days on a distributed file system and splitting prefill and decode across data centers. The biggest efficiency gain in LLM serving is no longer a faster kernel. It is turning recomputation into a storage lookup.

TL;DR For Engineers

DeepInfer is DeepSeek's Apache 2.0 production engine for DeepSeek-V4.1-Flash on NVIDIA SM100 and Ascend 950; offline inference only.
Not included: KV cache store, prefill/decode/vision disaggregation, and DP load balancers, which DeepSeek says reach ~80% of perfect balance.
Decode uses Wide-EP (EP64 to EP128); smallest efficient unit is 64 prefill + 64 decode GPUs.
SWA bounded replay recomputes 128 tokens after a cache hit; set DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140.
DSpark speculative decoding adds +5.4% at 60 seqs/rank but costs −1.5% at 16; tune for your concurrency.

Explain It Like I'm New

Running a popular AI chatbot is a bit like running a busy restaurant kitchen. The recipes (the AI model) matter, but whether you make money depends on how the kitchen runs: how orders are split between stations, how often cooks sit idle, and whether you can reuse prep work instead of starting every dish from scratch.

DeepSeek just shared the design of its kitchen. The software, called DeepInfer, is what runs its AI model for millions of users. It includes the fast cooking techniques, the way work is split across many machines, and a trick for guessing several words ahead and checking them in one go.

The most interesting part is about leftovers. When you send a long conversation to an AI, the system does a lot of work just reading it. DeepSeek saves that reading work on huge storage drives for up to a week, so if you come back, it can skip most of the effort.

What DeepSeek did not share is the system that manages those saved leftovers and routes customers between kitchens. Those parts are tied to its own infrastructure.

This matters because the cost of AI is increasingly decided by clever reuse and coordination, not just by faster chips or bigger models.

See It In Action

DeepSeek V4.1 Flash Inference Deployment Q&A, DeepSeek. github.com/deepseek-ai/DeepInfer/doc. The most detailed public description of a frontier lab's serving stack: parallelism choices, EPLB settings, KV storage tiers, TTLs, and speculative decoding data.
DeepInfer README and benchmarks, DeepSeek on GitHub. github.com/deepseek-ai/DeepInfer. Prefill and decode Pareto charts measured on 4 and 64 SM100 GPUs, plus run_sweep.sh to reproduce them.
DeepInfer tutorial, Agents Pulse. agentspulse.github.io. A practitioner walkthrough of building and running offline inference.
PagedAttention, vLLM team. arxiv.org/abs/2309.06180. The paged KV cache idea DeepInfer's block cache builds on, scaled here from GPU memory to a distributed file system.

Community Conversation

The New Stack, DeepSeek open sources the engine behind its API. Recalls that in April 2025 DeepSeek declined to release its engine because it was "tightly coupled with our internal infrastructure," and notes the company has not explained what changed.
r/LocalLLaMA, via The New Stack. One commenter credited DeepSeek's quant finance roots: "Once again Deepseek shows how good high-frequency traders are at optimizing performance."
MarkTechPost, DeepInfer breakdown. Reads the decode chart at above 25,000 tok/s/GPU and prefill near 90,000 non-cached tok/s/GPU, and compares scope with vLLM, SGLang, and TensorRT-LLM qualitatively.
AI Weekly, V4.1-Flash specs. Reports the 552B total, 8B prefill and 16B decode active parameters, and the claim of a 4x smaller KV cache than V4-Flash.
CellCog, what it takes to run. Focuses on the practical gap between the release and a production deployment.

— # (#)

Inference Economics Now Live in Storage, Not Kernels

DeepInfer is a generous release: production kernels, a zero idle scheduler, confidence scheduled speculative decoding, and a deployment Q&A more candid than anything a frontier lab has published about serving. But read it closely and the lesson is clear. The kernels make each GPU fast; the 100 PiB KV cache, the length tiered prefill balancer, and the KV aware decode balancer make the fleet cheap, and those stayed home. For most teams, the most valuable artifact here is not the code. It is the blueprint for the storage and scheduling layer they now know they need to build.

References

DeepSeek-V3 Technical Report, DeepSeek-AI. MLA, auxiliary loss free MoE balancing, and the multi token prediction that DSpark descends from.
Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. Paged KV cache, the foundation of block level caching.
Fast Inference from Transformers via Speculative Decoding, Leviathan, Kalman, Matias. The draft then verify scheme DSpark extends with confidence scheduling.
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, Zhong et al. The case for splitting prefill and decode onto separate resources.
You Only Cache Once: Decoder-Decoder Architectures for Language Models, Sun et al. The YOCO design behind projecting decoder KV from a single shared cache.

DeepInfer is DeepSeek's Apache 2.0 inference engine for DeepSeek-V4.1-Flash on NVIDIA Blackwell and Huawei Ascend 950, with wide expert parallelism, confidence scheduled speculative decoding, and constant cost cache replay. Its biggest lesson is what it leaves out: a 3 day, 100 PiB KV cache and the load balancers that make DeepSeek's 155 trillion tokens a day cheap.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀

Leave Granola and get up to 12 months free of Wispr Flow Notetaker + Dictation

If you have paid time left on an individual Granola plan, we'll match it with a Wispr Flow subscription that includes Notetaker and dictation, and add bonus time, up to 12 months total. Sign in or create a Wispr account and submit proof of your plan to check eligibility.

Try Notetaker Free