vLLM Jev Shows Decision Serving Is Won by Batching, Not Clever Scheduling

In partnership with

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 31, 2026

The promise: by the end of this issue you will understand how a decision model answers without generating a single token, why that turns serving into a prefix-caching problem, how RadixFlow patches vLLM's scheduler, and which of the project's speedups come from what.

The fence: we cover the vLLM Jev repository (158 commits, last on October 23, 2026), its RadixFlow code, benchmark tables and audit JSONs, the Open-Jev-2B checkpoint it uses for benchmarks, and the RadixFlow paper (read through its abstract only). We did not run the GPU benchmarks; our analysis decomposes the project's published numbers. Windows, Ascend and per-model adapter details are summarized, not dissected.

What It Actually Does

Decision models are LLMs trained to pick, not write. The idea took off with TypeSafe AI's Jev, a "System One" model released September 22, 2026. You send a state (the data) and typed questions; it returns structured answers with probabilities and zero output tokens. Three question types exist: Choice (one of N labels), Noul (probability a statement is true) and Score (a level on an ordered rubric). A wave of open checkpoints followed: Open-Jev, Tiny-Jev, Laya, Valen, Kev, Decider, Cloudflare's Clef and more.

vLLM Jev is a community serving layer for these checkpoints on top of vLLM. One command downloads a supported model, exports its decision head and serves the /v1/systemone API.

Property	Value
Engine	vLLM 0.29.0 pooling runner on Linux CUDA (Ascend via vLLM 0.22.1 bridge)
Other platforms	Apple Silicon (MLX / PyTorch MPS), experimental Windows CPU (PyTorch or pinned GGUF)
Models	18 main-table checkpoints plus 19 experimental adapters, up to Drex v2.0 at 35B
Inputs	Text and JSON; images for Valen, vjev, RSI-Jev v4, JevAny, Clef; short MP4 video for Valen and Clef
API	`POST /v1/systemone` with `state` and typed `questions`; `output_tokens: 0`
RadixFlow	Opt-in scheduler: bounded admission + state-ready traversal, Rust prefix index
Code	~20K lines of Python, ~1K for RadixFlow, a 97-line Rust index

The repo's headline speedups over the original authors' servers run from 2.3x to 43x. The RadixFlow increment over stock vLLM is separate, and much smaller.

The Architecture, Unpacked

Caption: A decision is a batch of prefill-only branches that share a prefix. Everything RadixFlow does happens in the scheduler box, deciding which branches enter each worker round.

Four design decisions matter most.

One: answer by prefill, not decoding. Each candidate becomes a token row ending at a readout position. The model runs one forward pass, and a small head scores the hidden state. No sampling, no output tokens, so latency is pure prefill.

Two: candidates share a prefix. Four candidates for one question repeat the same state and instructions. With native prefix caching, the shared part is computed once: 45% of tokens are reused on short RACE inputs and 84% on long QuALITY inputs.

Three: hybrid models break the caching assumption. Open-Jev builds on Qwen3.5, whose Gated DeltaNet layers keep a recurrent state rather than per-token KV. A branch can match a cached prefix whose state another branch is still computing in the same round. Stock vLLM then stops scanning its waiting queue.

Four: patch, don't fork. RadixFlow is a Scheduler subclass that rewrites the pinned vLLM 0.29.0 scheduler's source at startup and refuses to run on any other version.

The Code, Annotated

First, the decision request, verbatim from the README.

# The answer space is closed: the model can only return one of the criteria keys.
curl -sS http://127.0.0.1:8795/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"state":"I was charged twice and want a refund.","questions":{"intent":{"type":"choice","instructions":"Choose the customer intent.","criteria":{"billing":"A payment or refund issue","technical":"A malfunction or setup issue","other":"Another request"}}}}'
# Response: answers.intent.choice, answers.intent.probabilities, usage.output_tokens = 0  ← no decoding at all

Caption: The same schema powers TypeSafe's hosted Jev and, as we saw in our LFM3 issue, Liquid AI's d1. Typed decisions are becoming a shared interface.

Second, how a branch becomes a probability, from vllm_jev/decision.py (trimmed).

async def _encode(self, ids, salt, positions):
    async for output in self.engine_client.encode(
        prompt={"prompt_token_ids": ids, "cache_salt": salt},       # salt scopes prefix reuse
        pooling_params=PoolingParams(task="token_embed", use_activation=False),  # ← prefill only, no sampling
        request_id=request_id):
        final = output
    hidden = torch.as_tensor(final.outputs.data)[positions]          # keep only readout rows

def _probabilities(self, hidden, positions, count, kind):
    logits = F.linear(hidden[0], self.head["weight"][:count])        # tiny FP32 head over BF16 states
    temperature = self.compiler.manifest.get("temperatures", {}).get(kind, self.compiler.manifest["temperature"])
    return torch.softmax(logits / temperature, -1).tolist()

Caption: vLLM's pooling runner does the heavy lifting. A decision is "embed these rows and read a few positions," which is why continuous batching gives such large speedups over serialized author servers.

Third, RadixFlow's surgery on vLLM, verbatim from vllm_jev/radixflow/adapter.py (trimmed).

def adapt_schedule(original):
    tree = ast.parse(textwrap.dedent(inspect.getsource(original)))   # vLLM's own Scheduler.schedule source
    for node in ast.walk(tree):
        if not isinstance(node, ast.If):
            continue
        waiting_failure = ast.unparse(node.test) == "new_blocks is None" and isinstance(node.body[-1], ast.Break)
        if waiting_failure:
            node.body[-1] = ast.parse('''
if self._radixflow_reject(request, "allocation", num_scheduled_tokens):
    request_queue.pop_request()
    step_skipped_waiting.prepend_request(request)
    continue                                   # ← THIS is the trick: the native `break` becomes a local defer
else:
    break
''').body[0]
    ...
    if allocations != 1 or gates != 1 or zeroes not in (0, 2):
        raise RuntimeError("unsupported native scheduler structure")   # refuse unknown vLLM shapes

Caption: A global scan barrier becomes a per-request wait. Rewriting upstream source with the AST avoids a fork but ties the feature to one vLLM release, enforced by a version check.

Fourth, the five-millisecond admission rule, verbatim from vllm_jev/radixflow/policy.py (trimmed).

def snapshot(self, waiting, running, rows, budget, residual, now):
    peers = [r for r in waiting if r.request_id not in self.released]
    delay = self.config.wait_ms / 1000                                # default 5 ms
    oldest = min(self.arrivals.get(r.request_id, now) for r in peers)
    reason = "deadline" if now >= oldest + delay else None
    if len(peers) >= rows: reason = "rows"
    if self.config.policy in ("flex", "radixflow", "radixflow_frag"):
        if running: reason = "running"                                # never hold work behind a busy GPU
        elif sum(residual(r) for r in peers) >= budget: reason = "tokens"
    if reason:
        self.released.update(r.request_id for r in peers); return set()
    return {r.request_id for r in peers}                              # ← hold: let sibling branches arrive

Caption: The one-line idea behind the largest gain: an idle GPU waits briefly so a decision's candidate branches land in the same round.

Fifth, our decomposition of the published pressure tests.

# Open-Jev-2B, req/s: author → native vLLM → RadixFlow     (RF share of total log speedup)
# short C1:   6.45 →  7.53 → 11.54   +53.3%   (73%)   worker steps 80 → 48, PENDING recoveries: 0
# short C8:   6.42 → 35.57 → 38.91    +9.4%    (5%)
# short C32:  6.43 → 50.19 → 50.82    +1.3%    (1%)
# long  C8:   0.68 →  8.39 →  8.77    +4.5%    (2%)   steps −26%, ms/step +24%, host total −8.6%
# long  C32:  0.68 →  9.36 →  9.65    +3.1%    (1%)   steps −28%, ms/step +28%, host total −7.4%
# Open-Jev-9B long C8 / C32: +0.4% / +1.4%; long C1 on both models: −6.3% / −4.0%

Caption: From concurrency 8 up, stock vLLM batching supplies 95 to 100% of the speedup over author servers. RadixFlow's deferral trades fewer rounds for heavier ones.

It In Action

Input: the refund request from snippet one, served by Open-Jev-2B on one A800 with prefix caching, at concurrency 1.

Step one, compile. The compiler renders the state, the instructions and each criterion into token rows, one per candidate (four in the benchmark workload). All rows share the state-and-instructions prefix and differ only in the candidate tail. A fresh random cache_salt keeps reuse inside this request.

Step two, schedule (stock vLLM). Rows are admitted as they arrive. A sibling that matches a prefix whose recurrent state is still being produced in the current round cannot run yet, so the scan stops there and it waits for a later round. Over the 32-request short benchmark, that averages 2.5 worker rounds per decision (80 rounds).

Step two, schedule (RadixFlow). The GPU is idle, so admission holds the first row for up to 5 ms until its siblings arrive, then releases them together. The trace shows 1.5 rounds per decision (48 rounds) and zero PENDING deferrals: the gain is batching, not traversal.

Step three, score. Each row's readout hidden state goes through the FP32 head, then a softmax over the candidates.

Output: a Choice answer (billing) with per-label probabilities and output_tokens: 0. Mean HTTP latency was 132.5 ms on native vLLM vs 86.4 ms with RadixFlow, and the author's own server took 154.7 ms. All paired selections agreed across the 2B runs; probabilities differed by up to 0.06.

Open-Jev-2B, short input	Author server	Native vLLM	RadixFlow
Throughput, C1	6.45 req/s	7.53 req/s	11.54 req/s
Mean latency, C1	154.7 ms	132.5 ms	86.4 ms
Worker rounds, C1 (32 requests)	n/a	80	48
Throughput, C32	6.43 req/s	50.19 req/s	50.82 req/s

Caption: Project-reported, mean of three fresh services. At C1 the win is admission batching; at C32 stock vLLM is already 7.8x the author server and RadixFlow adds 1.3%.

Why This Design Works, and What It Trades Away

Why it works. Decision serving is a prefill workload with heavy prefix sharing and almost no decode, exactly where vLLM's continuous batching and paged prefix cache shine. Moving checkpoints from serialized author scripts onto vLLM's pooling runner yields 2.3x to 43x. RadixFlow then targets a real gap: hybrid recurrent state cannot be partly shared like KV blocks, so stock vLLM halts its scan at the first pending dependency. Deferring only that request keeps the batch full.

What it trades away.

Heavier rounds. On long inputs RadixFlow cuts worker rounds by 26 to 30%, but each round gets 24 to 38% more expensive on the host, leaving a net 0 to 5% throughput gain.
Single-stream regressions. Long inputs at concurrency 1 lose 6.3% (2B) and 4.0% (9B): the 5 ms hold costs more than it saves when nothing arrives to batch with.
Version lock. RadixFlow requires vLLM 0.29.0, synchronous scheduling, pipeline parallelism 1 and native prefix caching; async scheduling and remote KV connectors are rejected.
Numerical drift under load. In a 1,200-request concurrent stress test, Kev-0.8B produced 196 Choice answer flips with default BF16 matmuls, and still 96 to 98 with split-K disabled.
Full-attention models. On the OpenJev-0.6B control, RadixFlow regressed 12.2% at C8 and 6.9% at C32, and changed 15 of 288 paired answers.

Technical Moats

Adapter breadth. 37 decision checkpoints, each with its own prompt protocol, readout positions and head (linear, slot, pointer, letter), validated against the original authors' runtimes. The compatibility matrix is the moat.
Measurement discipline. Fresh-service triplicates, actual worker-call traces, host time per step, native cache-hit tokens and paired-answer agreement, with audit hashes in JSON. Few serving projects publish their own regressions this plainly.
Cross-platform reach. The same CLI and API on CUDA, Huawei Ascend 910B3, Apple Silicon and Windows CPU (24 checkpoints with four isolated dependency profiles).
Safe upstream patching. AST rewriting with structural assertions, a source hash in telemetry and explicit rejection of unsupported configurations lets a small team modify vLLM's scheduler without maintaining a fork.

Contrarian Insights

Insight One: the RadixFlow headline is admission batching wearing a traversal label. The +53.3% (2B) and +41.4% (9B) gains occur at short inputs and concurrency 1, where the traces record zero PENDING recoveries. The repo says so itself: the gain "includes admission/pooling adaptation" and "is not attributed to state-ready traversal alone." In the cells where deferral actually fires, long inputs at C8 and C32, the throughput gain is +0.4% to +4.5%. The paper abstract reports 3 to 24% lower latency across five hybrid models with most gain from traversal on larger Qwen3.5 models, so the two evaluations measure different things. For practitioners, the actionable idea is cheap: a few milliseconds of admission delay on an idle server to batch sibling branches. Any serving stack can do that.

Insight Two: decision models make nondeterminism a correctness bug. A chatbot that rounds differently under load writes a slightly different sentence. A decision model that rounds differently returns a different label. Batch-dependent BF16 rounding flipped Kev-0.8B's Choice answer 196 times in a 1,200-request stress test; the repo also flags Score-level changes for This-That and Decider-2B under concurrent batching. Many decisions sit near ties (OpenJev-0.6B's changed answers had top-two margins of 0.004 to 0.02). If you route tickets, approve refunds or gate actions on argmax, log the probability margin and treat near-ties as uncertain. Faster batching makes this worse, not better.

Surprising Takeaway

The fastest change in vLLM Jev's scheduler is to wait. At concurrency 1 on short inputs, holding an idle GPU's first request for up to 5 ms until its candidate branches arrive cut worker rounds from 80 to 48 and host time by 40%, which is the entire +53% headline. The more sophisticated dependency deferral, the part that needed AST surgery on vLLM, contributes at most 5% at the concurrencies where production systems actually run. And at those concurrencies, stock vLLM is already 4x to 14x faster than the original authors' servers.

TL;DR For Engineers

vLLM Jev serves typed decision models (Choice, Noul, Score) on vLLM's pooling runner: candidates become prefill-only rows scored by a small head, with zero output tokens.
Most of the speedup over author servers (2.3x to 43x) is plain vLLM continuous batching plus prefix caching (45 to 84% token reuse).
RadixFlow's +53% peak comes from a 5 ms admission hold at concurrency 1; its dependency deferral adds 0 to 5% at C8 and C32 and regresses long single-stream inputs.
RadixFlow is pinned to vLLM 0.29.0 synchronous scheduling and AST-patches the scheduler; keep it off unless your workload matches the tested cells.
Decision answers flip under batch-dependent BF16 rounding. Log probability margins and handle near-ties explicitly.

Explain It Like I'm New

Most AI models answer by writing. Ask whether an email is spam and a chatbot types a sentence, which software then has to read and interpret. That is slow and can be messy.

Decision models skip the writing. You hand them some information and a multiple-choice question, and they return a probability for each allowed answer: billing 0.93, technical 0.04, other 0.03. Code can act on that instantly.

Under the hood, the model checks each possible answer by reading the information plus that answer. Since every answer starts with the same information, a smart server reads the shared part once and reuses it, like a teacher grading four versions of a test that share the same first page.

vLLM Jev is that smart server for a growing family of these models. It also experiments with a new way of ordering the work for models that keep a running memory, which cannot always be reused the moment it is needed.

The lesson from its own measurements is that the simplest ideas did most of the work: grouping requests together and reusing shared text. As AI becomes a component inside software making millions of small decisions, cheap, fast and consistent answers will matter more than eloquent ones.

See It In Action

RadixFlow explainer video, vLLM Jev maintainers. Watch on GitHub. An animated walkthrough of the pending-state dependency and how deferral keeps a worker round full.
Valen video decisions, live comparison, vLLM Jev. MP4. One checkpoint answers 24 structured questions about an archery clip: 5.31 s on the PyTorch reference vs 1.22 s on vLLM Jev, all answers matching.
Apple Silicon maze decisions, vLLM Jev. MP4. Valen 2B answering GameQA maze questions locally on an M5 through MLX.
Road Decider demo, Liquid AI cookbook. GitHub. A game where hosted Jev and Liquid's d1 drive cars through the same decision API several times per second.

Community Conversation

TypeSafe AI, Jev model listing. Positions Jev as a "System One" model for bounded judgments, at $0.042 per million input tokens with free output and 70 to 500 ms vendor-reported latency.
Hora de Codar, Choice, Score or Noul. A practitioner guide to which Jev question type fits which decision, a sign the interface is spreading beyond its origin.
HackerNoon, Comparing Jev and LLMs on structured decision tasks. An independent comparison of decision models against general LLMs on structured tasks.
vLLM Jev contributors, repository pull requests. Community adapters for RSI-Jev, Clef, JevAny and a BF16 precision mode, plus the first PR on shared-prefix reuse that led to RadixFlow.

Batch First, Then Get Clever

vLLM Jev is excellent infrastructure for an interface that is quietly becoming standard: typed decisions with probabilities and no prose. Its own measurements tell a clear story. Putting decision checkpoints on vLLM's batching and prefix cache delivers multiples; a few milliseconds of admission delay delivers the headline; the elegant dependency deferral delivers single digits and only on hybrid models under load. Adopt the serving layer, borrow the admission idea, leave RadixFlow off until your workload looks like the cells where it wins, and budget engineering time for the bigger risk: answers that flip when batches change.

References

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving, Jing et al. State-ready traversal for hybrid models; 3 to 24% lower latency and 11 to 44% fewer worker rounds across five models.
Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. The vLLM paged KV cache that makes prefix sharing cheap.
SGLang: Efficient Execution of Structured Language Model Programs, Zheng et al. RadixAttention, the radix-tree prefix cache that inspired RadixFlow's name.
Marconi: Prefix Caching for the Era of Hybrid LLMs, Pan et al. Why recurrent state changes prefix caching for hybrid attention and SSM models.

vLLM Jev serves typed decision models on vLLM's pooling runner, scoring candidates by prefill with zero output tokens and gaining 2.3x to 43x over author servers mostly from batching and prefix caching. Its experimental RadixFlow scheduler peaks at +53% through a 5 ms admission hold, while its hybrid-state deferral adds 0 to 5% under load.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀

From invisible to first in ChatGPT.

Bob's Your Uncle has spent twenty years helping challenger brands win their categories. But when buyers asked ChatGPT about their own category, the agency didn't show up.

With ChatFeatured, their AI visibility went from 7% to 22% in 90 days, and they went from invisible to first in ChatGPT.

They're not the only ones:

Vessel: AI visibility up 2,200%+

SCOJO: AI visibility up 160%

WikiSleep: brand mentions up 113%

ChatFeatured finds the questions where AI leaves you out, writes the content that gets you cited, and reports the lift every week. Your team decides what ships.

Most customers see their first citations inside 7 days.

Get recommended by AI