vLLM Semantic Router: The Infrastructure Layer That Decides Which Model Should Handle Your Request Before the Model Sees It

git clone https://github.com/vllm-project/semantic-router
cd semantic-router

# Development:
pip install -e ".[dev]"

# Production (Docker Compose):
docker compose -f deploy/docker-compose.yml up -d
# → starts router API, dashboard, backend registry

# vLLM Semantic Router: signal extraction and Boolean decision composition
# Source: vllm-project/semantic-router, config/ + router/ directories
# Reconstructed from arXiv:2603.04444 architecture description

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
import time

class SignalTier(Enum):
    HEURISTIC = "heuristic"   # sub-millisecond, no model inference
    NEURAL    = "neural"      # milliseconds, requires embedding/classifier

@dataclass
class Signal:
    name: str
    tier: SignalTier
    value: any              # bool, str, float, or categorical
    latency_ms: float       # how long extraction took

@dataclass
class SignalVector:
    """All signals extracted from a single request."""
    # Tier 1: heuristic (always extracted, sub-millisecond)
    keyword_match: Optional[list[str]] = None  # matched blocklist terms
    language:      Optional[str]       = None  # ISO 639-1 code
    context_length: Optional[int]      = None  # token count
    role:          Optional[str]       = None  # "public", "premium", "admin"

    # Tier 2: neural (extracted conditionally)
    domain:        Optional[str]       = None  # "medical", "coding", etc.
    embed_sim:     Optional[float]     = None  # similarity to route exemplars
    modality:      Optional[str]       = None  # "text", "image", "audio"
    factual_density: Optional[float]  = None  # 0-1, triggers HaluGate if high

    extraction_total_ms: float         = 0.0


class SignalExtractor:
    """
    Extracts all signals from an incoming request.

    Two-tier design rationale:
    ← Tier 1 (heuristic) runs on EVERY request with no GPU cost.
      If heuristic signals are sufficient to make the routing decision, stop here.
      No embedding model is invoked, no classifier runs.

    ← Tier 2 (neural) runs CONDITIONALLY: only when heuristic signals
      are insufficient for a definitive decision.
      This is the Shannon-inspired "information maximization" view:
      extract the minimum signal needed for the routing decision.
    """

    def __init__(self, embedding_model, domain_classifier):
        self.embed_model = embedding_model   # mmBERT-embed-32k-2d-matryoshka
        self.domain_clf  = domain_classifier

    def extract(self, request: dict, run_neural: bool = True) -> SignalVector:
        t0 = time.monotonic()
        sv = SignalVector()

        # ── TIER 1: HEURISTIC (sub-millisecond) ──────────────────────────────
        # Keyword matching: scan prompt text for blocklist patterns
        # ← No tokenization needed: simple string scan
        sv.keyword_match = self._keyword_scan(request["messages"])

        # Language detection: FastText-based, ~0.1ms per request
        sv.language = self._detect_language(request["messages"])

        # Context length: count tokens (or estimate from char count)
        sv.context_length = self._count_tokens(request["messages"])

        # Role-based authorization from request headers / API key scope
        sv.role = request.get("x_caller_role", "public")

        # ← Decision: if heuristic signals alone determine routing, skip neural
        # Example: PII keyword match → ROUTE_LOCAL_ONLY immediately
        if sv.keyword_match and "PII" in sv.keyword_match:
            sv.extraction_total_ms = (time.monotonic() - t0) * 1000
            return sv  # ← skip neural tier entirely

        # ── TIER 2: NEURAL (conditional) ─────────────────────────────────────
        if run_neural:
            # Embed the request for domain and similarity routing
            embedding = self.embed_model.encode(request["messages"])
            # ← 2D Matryoshka: embedding is usable at any prefix length
            #   [0:128] for fast routing, [0:768] for precision routing
            sv.embed_sim = self._compute_similarity(embedding)
            sv.domain    = self.domain_clf.predict(embedding)
            sv.modality  = self._detect_modality(request)
            sv.factual_density = self._estimate_factuality(embedding)

        sv.extraction_total_ms = (time.monotonic() - t0) * 1000
        return sv


class DecisionEngine:
    """
    Composes signals into routing decisions via Boolean rules.

    ← The rules are compiled at LOAD TIME (not request time).
      Conflict detection happens during compilation:
      if rule A says "route to local" and rule B says "route to cloud"
      and both can fire on the same request, the engine flags it.
      No surprises in production.

    ← The same engine with different rule configs produces different
      deployment behaviors: cost-optimized, privacy-regulated, etc.
    """

    def __init__(self, rules: list[dict]):
        self.compiled_rules = self._compile(rules)
        # ← Conflicts detected here, not at request time
        self._check_conflicts(self.compiled_rules)

    def decide(self, sv: SignalVector) -> list[str]:
        """Evaluate all rules and return list of triggered decisions."""
        decisions = []
        for rule in self.compiled_rules:
            if rule.evaluate(sv):
                decisions.append(rule.decision)
        return decisions


# ── EXAMPLE CONFIGS FOR TWO DIFFERENT DEPLOYMENTS ────────────────────────────
# Same architecture, different rules = different behavior

COST_OPTIMIZED_RULES = [
    # Route to cheap model unless domain is medical/legal (risk-sensitive)
    {"if": "domain NOT IN ['medical', 'legal'] AND context_length < 8000",
     "then": "ROUTE_EFFICIENT_MODEL"},
    {"if": "context_length >= 8000",
     "then": "ROUTE_LONG_CONTEXT_POOL"},
]

PRIVACY_REGULATED_RULES = [
    # ← ALL requests stay on-premise: no cloud routing allowed
    {"if": "keyword_match CONTAINS 'PII' OR role != 'premium'",
     "then": "ROUTE_LOCAL_ONLY"},
    {"if": "modality == 'text'",   "then": "ROUTE_LOCAL_VLLM"},
    {"if": "modality == 'image'",  "then": "ROUTE_LOCAL_VISION"},
]
# ← Same signal extraction, different Boolean rules, no code changes

# vLLM Semantic Router: deployment configuration
# Source: vllm-project/semantic-router/config/ directory structure
# Shows how signal → decision → selection → plugin chain composes in YAML

# ─── Signal configuration ───────────────────────────────────────────────────
signals:
  heuristic:
    keyword_match:
      enabled: true
      blocklist_path: ./config/blocklists/pii_terms.txt
      categories: ["PII", "JAILBREAK", "COMPETITOR"]
    language_detect:
      enabled: true
      backend: fasttext
      min_confidence: 0.85
    context_length:
      enabled: true
      estimator: tiktoken
      model: gpt-4o  # use for consistent token counting across backends

  neural:
    domain_classifier:
      enabled: true
      # ← Only runs when heuristic signals don't resolve the decision
      model: ./models/domain-classifier-v2
      threshold: 0.7
    embedding_similarity:
      enabled: true
      # 2D Matryoshka model: use prefix of 128 dims for fast routing
      model: llm-semantic-router/mmbert-embed-32k-2d-matryoshka
      dim: 128        # ← use 128-dim prefix for routing, not full 768
      # ← 98x speed improvement (arXiv:2603.12646) via Flash Attention
      #   + prompt compression + near-streaming embedding

# ─── Decision rules ─────────────────────────────────────────────────────────
decisions:
  - name: route_medical_premium
    condition: "domain == 'medical' AND role == 'premium'"
    target_pool: frontier_pool
    plugins: [pii_filter, halugate_medical]

  - name: route_local_privacy
    # ← Privacy: PII or public role → never leave local infrastructure
    condition: "keyword_match CONTAINS 'PII' OR role == 'public'"
    target_pool: local_vllm_pool
    plugins: [pii_redact, pii_restore]

  - name: route_long_context
    condition: "context_length >= 32000"
    target_pool: long_context_pool
    selection_algorithm: latency_optimal  # ← override default for long context

# ─── Backend pools ───────────────────────────────────────────────────────────
pools:
  local_vllm_pool:
    backends:
      - type: vllm
        url: http://localhost:8000
        model: Llama-3.1-70B-Instruct
      - type: vllm
        url: http://localhost:8001
        model: gemma-4-12b-it

  frontier_pool:
    backends:
      - type: openai
        model: gpt-4o
      - type: anthropic
        model: claude-opus-4-6
      - type: bedrock
        model: us.amazon.nova-pro-v1

  long_context_pool:
    backends:
      - type: openai
        model: gpt-4o-128k
      - type: gemini
        model: gemini-2.0-flash

# ─── Plugin definitions ──────────────────────────────────────────────────────
plugins:
  halugate_medical:
    # 3-stage hallucination detection for medical domain
    # ← Medical + factual claims → run full HaluGate pipeline
    stages:
      - pre_check:    {method: factual_density, threshold: 0.6}
      - consistency:  {method: source_grounding, retrieval: pubmed_rag}
      - post_check:   {method: factuality_verify, backend: gpt-4o-mini}
    # ← If any stage fails: return uncertainty flag in response metadata
    on_failure: flag_response  # not block: medical decisions need human review

  pii_redact:
    method: presidio           # Microsoft Presidio for PII detection
    entities: [PERSON, EMAIL, PHONE, SSN, CREDIT_CARD]
    action: pseudonymize       # replace with consistent placeholder
    # ← pii_restore plugin maps placeholders back in the response

{
  "messages": [
    {"role": "user", "content": "What is the recommended dosage of metformin 
     for a patient with John Smith (DOB 1965-03-12) and mild renal impairment?"}
  ],
  "x_caller_role": "premium",
  "x_request_id": "req_7f3a91b"
}

Tier 1 (heuristic): 0.8ms
  keyword_match:    ["PII"]   ← "John Smith" + DOB detected as PII
  language:         "en"
  context_length:   48 tokens (short)
  role:             "premium"

Tier 2 (neural): 3.4ms  ← runs because role=premium, not blocked by PII alone
  domain:           "medical" (confidence: 0.94)
  factual_density:  0.87      ← high: dosage + disease = factual claims
  modality:         "text"

Rule: route_medical_premium
  Condition: domain == 'medical' AND role == 'premium'
  Evaluation: TRUE
  Result: → frontier_pool + [pii_filter, halugate_medical]

Rule: route_local_privacy
  Condition: keyword_match CONTAINS 'PII' OR role == 'public'
  Evaluation: TRUE (PII detected)

⚠ CONFLICT DETECTED at load time for requests matching BOTH rules:
  premium medical request with PII → fires both route_medical_premium AND route_local_privacy
  Resolution: priority ordering in config (privacy > frontier for PII + premium medical)
  → ROUTE_LOCAL_VLLM + [pii_redact, halugate_medical, pii_restore]
  ← Conflict caught at config load, not at runtime. PII wins over frontier routing.

1. pii_redact (pre-generate):
   "John Smith" → "PERSON_1"
   "DOB 1965-03-12" → "DOB_1"
   Redacted request forwarded to local vLLM

2. Local vLLM inference (Llama-3.1-70B-Instruct):
   Input: "...dosage of metformin for PERSON_1 (DOB_1) and mild renal impairment?"
   Output: "For patients with mild renal impairment, metformin 500mg twice daily..."
   Inference time: ~1.8s

3. halugate_medical (post-generate):
   Stage 1 (density): factual_density=0.87 → proceed to stages 2+3
   Stage 2 (consistency): checks "500mg twice daily" against PubMed RAG
     → confirms: consistent with KDIGO 2022 guidelines for eGFR 45-60
   Stage 3 (factuality): gpt-4o-mini verification
     → confidence: 0.91  → PASS
   Metadata added: {halugate_verified: true, sources: ["KDIGO2022", "FDA_label_metformin"]}

4. pii_restore:
   "PERSON_1" → "John Smith"
   "DOB_1" → "DOB 1965-03-12"

Total routing overhead: 4.2ms (signal) + 12ms (plugins)
Model inference: 1.8s
Total request latency: ~1.82s
Backend used: local_vllm_pool (not frontier, due to PII policy)

Without Semantic Router:
  PII data sent to cloud endpoint (policy violation)
  No hallucination detection on medical dosage advice
  Routing logic: if-else in application code

With Semantic Router:
  PII stays local (policy enforced at routing time)
  Medical claim verified against clinical literature (3-stage HaluGate)
  Policy conflict (PII + medical-premium) detected at config load
  Routing logic: declarative YAML, auditable, conflict-free