vLLM Semantic Router does not only forward prompts to one LLM. It classifies and routes requests using signals — keywords, classifiers, and embeddings.

Embeddings turn text into vectors. The router then:

• matches a query to category or anchor prompts (routing signals), or
• decides whether a new query is similar enough to reuse a semantic cache hit.

A general-purpose embedder (for example MiniLM) is good on average, but weak on specialized language (medical, law, code). So the project fine-tunes embeddings for those jobs, then publishes them on Hugging Face for everyone to download.

Training does not live inside the Go router. It lives under src/training/model_embeddings/ (Python). The router later loads the published models from the llm-semantic-router org.

End-to-end picture​

Domain data (queries / Q&A)
        ↓
Build training triplets (anchor, positive, negative)
        ↓
Fine-tune base embedding model (LoRA or full)
        ↓
Evaluate (margin / MRR / recall)
        ↓
Upload to Hugging Face (llm-semantic-router/...)
        ↓
Users / CI download model
        ↓
Semantic Router uses it for routing or cache similarity

There are two main pipelines.

Pipeline A — Cache embedding LoRA​

Path: src/training/model_embeddings/cache_embeddings/

Job: Improve semantic cache accuracy so “same intent” hits and “related but different” misses.

Step 1 — Collect unlabeled queries​

Prepare a JSONL file with one query per line for a domain (for example medical questions). See domains/README.md.

Step 2 — Generate triplets with an LLM​

generate_training_data.py (optionally via vLLM) builds contrastive samples:

That teaches: paraphrases should be close; near-misses should stay apart — exactly what a cache needs.

python3 src/training/model_embeddings/cache_embeddings/generate_training_data.py \
  --input data/cache_embeddings/medical/unlabeled_queries.jsonl \
  --domain medical \
  --output data/cache_embeddings/medical/triplets.jsonl \
  --max-queries 10000

Step 3 — Train a LoRA adapter​

lora_trainer.py fine-tunes a small LoRA on a base such as sentence-transformers/all-MiniLM-L12-v2.

• LoRA is small (on the order of hundreds of KB), not a full retrain.
• Training uses contrastive / MNR-style loss on triplets.

python3 src/training/model_embeddings/cache_embeddings/lora_trainer.py \
  --train-data data/cache_embeddings/medical/triplets.jsonl \
  --base-model sentence-transformers/all-MiniLM-L12-v2 \
  --output models/cache/medical-cache-lora \
  --epochs 1

You can also train a multi-domain adapter by concatenating domain triplet files and training once.

Step 4 — Evaluate​

Primary cache metric:

margin = avg(similarity to positives) − avg(similarity to negatives)

Higher margin means better separation and fewer false cache hits.

python3 src/training/model_embeddings/cache_embeddings/evaluate_multi_domain.py \
  --lora-path models/multi-domain-cache-lora-L12 \
  --sample-size 2000

Step 5 — Push to Hugging Face​

After training (also supported by train-domain.sh):

cd models/cache/medical-cache-lora
huggingface-cli upload llm-semantic-router/<model-name> . --repo-type model

Published examples:

• multi-domain-cache-lora-L12
• medical-cache-lora
• Dataset: cache-embedding-test-sets

Step 6 — Consume​

Download the adapter, attach it to the base MiniLM with PEFT, and encode queries for cache similarity:

from sentence_transformers import SentenceTransformer
from peft import PeftModel

base_model = SentenceTransformer("sentence-transformers/all-MiniLM-L12-v2")
base_model[0].auto_model = PeftModel.from_pretrained(
    base_model[0].auto_model,
    "llm-semantic-router/multi-domain-cache-lora-L12",
)
embedding = base_model.encode("What are the symptoms of diabetes?")

Pipeline B — Domain-adapted full embeddings​

Path: src/training/model_embeddings/domain_adapted_embeddings/

Job: Better retrieval / domain routing vectors (for example medical), not only cache LoRA.

Inspired by Distilling an LLM’s Wisdom….

Step 1 — Prepare data​

prepare_data.py loads a Hugging Face dataset or custom Q&A JSON (question → answering passage).

python prepare_data.py \
  --source huggingface \
  --dataset keivalya/MedQuad-MedicalQnADataset

Step 2 — Iterative hard-negative mining​

1. Embed with the current model.
2. Mine hard triplets (true docs ranked low; wrong docs ranked high).
3. Keep some easy triplets so the model does not forget.
4. Fine-tune with TripletLoss (margin ≈ 0.1).
5. Repeat about two iterations, accumulating triplets.

Base model is often mmbert-embed-32k-2d-matryoshka.

python train.py \
  --data-dir data \
  --output-dir models/trained \
  --base-model llm-semantic-router/mmbert-embed-32k-2d-matryoshka \
  --iterations 2 \
  --learning-rate 5e-5 \
  --margin 0.1

Critical knobs: learning rate 5e-5, margin 0.1, easy:hard ratio about 2:1, and accumulate triplets across iterations.

Step 3 — Measure gains​

On MedQuAD, MRR@5 improved about 71% versus baseline after two iterations. The published model is mmbert-embed-medical.

Step 4 — Upload the full model to Hugging Face​

Same pattern: local models/trained/best → huggingface-cli upload to llm-semantic-router/....

How Hugging Face fits (publish + pull)​

Training is continuous in spirit: better data or hyperparameters → new fine-tune → new Hub upload.

How this reconnects to the running router​

User prompt
  → Embedding model (from HF / local cache)
  → Vector similarity vs anchors / cache entries
  → Signal(s)
  → Decision (AND/OR rules)
  → Algorithm (confidence, fusion, …)
  → Backend LLM

We do not train the chat LLM for routing. We train small embedding models so the router’s similarity signals stay accurate for real domains, then ship those models on Hugging Face as shared community assets.

Try it yourself​

One-line summary​

vLLM Semantic Router fine-tunes embedding models with contrastive/triplet learning on domain triplets, uploads them to the llm-semantic-router Hugging Face org, and downloads them at runtime so routing and semantic cache similarity stay accurate.

In less than a year since its public launch, vLLM Semantic Router has reached 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. Across three major releases—Iris, Athena, and Themis—the system boundary moved from choosing a model, to governing multi-model inference, to preserving state and coordination across sessions. Those releases built the foundation for the MoM architecture envisioned from day 0.

This post describes the next step for vLLM Semantic Router: moving from routing among models to building dependable model systems from them. Under one versioned contract, independent models, policies, preferences, and execution paths become a system that can be trained, evaluated, exported, imported, deployed, and invoked through one interface. Our goal is to make vLLM Semantic Router a training, evaluation, and inference engine for Mixture-of-Models.

Figure 1: A Mixture-of-Models turns a heterogeneous model portfolio into one model experience.

How vLLM-SR Got Here​

The first vLLM Semantic Router post asked a practical question: why give simple and difficult requests the same reasoning budget? A lightweight classifier used fixed domain labels to choose between fast and reasoning paths, helping vLLM spend inference compute more selectively.

Production traffic quickly exposed the limit of that design. Domain alone could not represent privacy, safety, context, language, modality, tools, preferences, latency, and authorization. A static label also could not account for an endpoint that was cheap but overloaded, capable but remote, or unsafe to switch into midway through an agent session.

We rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture that separated observed evidence from policy and execution. This became the spine of the next three releases.

Figure 2: Each stage changed the unit of control: model, decision, system, session, and finally the complete model lifecycle.

Iris made routing composable. Domain, keyword, embedding, factuality, feedback, and preference signals fed explicit decisions, while safety, PII protection, caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as “System Level Intelligence for Mixture-of-Models.”

Athena added first-class model selection, memory and RAG, a multilingual and multimodal model stack, ROCm acceleration, and an operating dashboard. The project was becoming the control system around multi-model inference, not just a classifier in front of vLLM.

Themis turned that broader system into an operable contract:

Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.

Themis added session-aware agentic routing, replayable traces, stronger protocol support, an operator console, and runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It also made a route explainable: operators can see the evidence, policy, algorithm, and physical model behind each decision.

From Signal–Decision to Workload–Router–Pool​

The releases built the runtime. Two project papers explained the architecture behind it.

The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. Fast heuristics and learned classifiers turn prompts, context, identity, safety, and modality into a structured signal vector; a Boolean engine then composes those signals into auditable policy. A typed neural-symbolic DSL parses and validates that policy before compiling it into deployable configuration. When the paper was published, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugins for caching, RAG, memory, safety, provider handling, and response validation.

The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:

• Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
• Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
• Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management

Those variables cannot be optimized independently. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project's research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.

Figure 3: The white paper defines the programmable routing engine; the vision paper connects it to workload and physical pool design.

Together, the papers made routing programmable and tied it to workload and hardware—the two foundations MoM brings under one model contract.

Meanwhile, the runtime was already moving beyond single-model selection. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.

Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.

Why the Model Boundary Has to Move​

Today's AI stack is fragmented along four axes:

• Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, and multimodal models will coexist. None wins simultaneously on quality, cost, latency, trust, privacy, and domain fit.

• Compute is fragmented. GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters differ in memory, kernels, availability, price, and energy use. Model choice and placement are becoming the same decision.

• Location is fragmented. Inference spans cloud, data center, and edge. Privacy or residency may rule out a stronger remote model, while a local workload may still need an on-demand cloud expert.

• Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, safety, style, and multimodality. Those choices should shape execution directly.

Today, each application has to reconcile these fragments on its own.

Figure 4: Before MoM, fragmented intelligence becomes application-side routing glue.

Mixture-of-Models moves that responsibility behind one model boundary.

At that boundary, intelligent allocation becomes part of the model. The engine determines which models are eligible, where execution can run, whether models should collaborate, and how to satisfy hard constraints.

Energy makes allocation inseparable from efficiency. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. The allocation layer controls demand: which work deserves those tokens, and which model or collaboration can provide them within the required quality, latency, and energy budget.

The application selects one versioned model identity and receives one attributable response. Its physical realization can still span open and closed models, cloud and edge, and different accelerator generations. The fragmentation remains, but it becomes internal to the model system instead of leaking into every application.

Figure 5: With MoM, the same fragmented resources become the internal realization of one model.

What We Mean by Mixture-of-Models​

A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.

A multi-upstream gateway can forward traffic without owning system quality. An MoM owns an objective, an evaluation contract, a reproducible composition, and the runtime that executes it.

MoM also differs from Mixture-of-Experts. MoE routes tokens among internal experts during one forward pass; MoM coordinates independent models that may differ in architecture, owner, license, modality, protocol, context window, and hardware. An MoE checkpoint can itself be one MoM component.

Figure 6: Selection is one MoM topology. Cascades, parallel fusion, and bounded workflows share the same model boundary.

A portable MoM therefore needs more than weights and configuration: it needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.

Open checkpoints can travel with the artifact; closed models remain authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable. It makes the model system reproducible.

Turn Preferences into Models​

Preferences become concrete when they are published as model identities. One MoM family can offer several operating points:

Each name is a versioned model contract, not a router preset. The application chooses the behavior it needs; vLLM-SR selects and coordinates the models that deliver it while preserving hard privacy, residency, authorization, and safety constraints.

To an application, the full system remains an ordinary model call:

{
  "model": "vllm-sr/mom-v1-ultra",
  "messages": [
    {"role": "user", "content": "Review this design and identify its weakest assumption."}
  ]
}

That identity may select one model, escalate through a cascade, compare parallel answers, require grounding, or run a bounded workflow—without changing the external interface, version, or response contract.

Figure 7: Preferences are published as bounded, versioned model contracts—not hidden application-side routing presets.

Four planes separate ownership:

Figure 8: A complete MoM spans four planes: artifact, learning, execution, and physical realization.

A deployment must map logical requirements onto the models and machines available in its environment. The proposal uses four objects:

1. The bundle fixes the interface, graph, policies, behavior variant, bounds, and immutable semantic assets.
2. The binding maps logical components to eligible deployments without changing the model's decision semantics.
3. The resolution lock freezes the constituent revisions, runtimes, images, accelerators, and provider observations.
4. The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.

Figure 9: One stable model identity, from portable contract to attributable run.

This separation keeps portability honest. The same mom-v1-ultra can bind to ROCm, CUDA, a private CPU or NPU node, or a hybrid deployment without promising identical outputs from opaque providers. Instead, it preserves control semantics, exposes substitutions, and gives serving and evaluation the same resolved system.

vLLM-SR as the MoM Engine​

Training, evaluation, and inference must share one contract; otherwise research, benchmarks, and production drift into different systems.

Training allocation, not only weights​

MoM training covers router-owned embeddings, signal encoders, preference and safety models, and selectors. It also learns allocation and collaboration: which path fits a workload and budget, when a cascade should stop, how a panel should judge or synthesize, and when an agent session should switch models. Because constituents may be independent or closed, progress does not require gradients through all of them; policies, thresholds, pools, prompts, contracts, and topology can be optimized from traces and outcomes.

The target is a frontier across quality, latency, cost, safety, privacy, reliability, locality, and energy. Replay and outcomes feed production experience back into offline training without letting the hot path silently rewrite policy.

Evaluating the MoM as one model​

Evaluation must score the model identity end to end; backend benchmarks are inputs, not the result. A versioned scorecard should measure routing regret, collaboration gain, recovery, session continuity, tail latency, cost, safety, privacy, and energy. It should stress provider failures, device loss, model disagreement, workload drift, and preference changes. Each declared operating point also needs its own test: flash on its latency–quality frontier, light against its quality floor, and ultra within its budget.

The scientific test is stricter than asking whether more calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Evaluations must report calls, tokens, cost, latency, and energy alongside quality—and publish when composition does not help.

Figure 10: Composition gain is meaningful only under matched active compute, with quality reported alongside calls, tokens, cost, latency, and energy.

Executing intelligence at inference time​

At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback, trace, and response contract; the application makes a normal model call.

Figure 11: MoM is a closed lifecycle: train the allocation policy, evaluate the full system, execute it, and turn outcomes into the next validated version.

One Model That Can Move​

Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. A logical specification compiles into an immutable bundle, binds to an environment, resolves the concrete deployment, and retains the same identity for serving and evaluation.

The artifact should run across developer machines, private clusters, cloud fleets, and edge environments while its physical realization changes. A specialist may resolve to an admissible local checkpoint or managed endpoint; an accelerator runtime may be replaced. If privacy makes a remote expert unavailable, the engine follows a declared fallback or abstention path. A binding cannot silently rewrite the graph, relax a guard, or turn a panel into a cascade—those changes require a new model version.

“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already supports paths across ROCm, CUDA, OpenVINO, and CPU. Next, hardware capability and placement become part of the MoM contract, allowing the engine to map the model system onto what is available.

The standard for the user experience is simple:

One model identity. Many models. Any hardware.

Figure 12: One logical model identity can be realized across developer, data-center, cloud, and edge hardware.

If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.

What Changes Now​

The next stage focuses on four connected areas:

1. Define a portable MoM specification. Package components, objectives, policy, preferences, evaluation, constraints, and execution semantics as one versioned artifact.
2. Close the training–evaluation–inference loop. Improve models and recipes from evaluation and replay, then ship them through reviewable, rollback-safe releases.
3. Build a heterogeneous runtime. Map one MoM across cloud, data center, and edge using hardware, locality, energy, and data boundaries as inputs.
4. Keep the model interface boring. Make an MoM as easy to import, deploy, and invoke as a single model.

Figure 13: Four connected workstreams turn Mixture-of-Models from an execution pattern into the next model architecture.

This is a research program for how independent models should specialize, compete, verify, and collaborate; how to measure the resulting system; and how one model contract can survive across devices and environments. Our mission is:

Advancing the science of intelligence across models, devices, and environments.

We will study when composition produces capabilities beyond a single checkpoint, treat placement and energy as part of intelligence, and carry the same model contract from edge to cloud and from research to production.

Build It With Us​

Building Mixture-of-Models requires more than routing. The work spans model training, evaluation, serving systems, hardware, and production operations.

Iris, Athena, and Themis improved because contributors brought real workloads, added backends, trained models, published benchmarks, found failure cases, and argued for better interfaces. MoM needs the same range of work: learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.

If you work on these problems, we want to learn from your workloads and measurements. Build an operating point, add a runtime, test a collaboration recipe, or publish a case where composition fails. MoM will be stronger if its assumptions are tested in the open.

Acknowledgments​

vLLM-SR has grown through work across engineering, research, and the wider ecosystem. We thank Xunzhuo Liu, Huamin Chen, Bowei He, Yankai Chen, Fuyuan Lyu, and Steve Liu for helping shape its technical and research direction. We also thank Andy Luo and Haichen Zhang for their work on ROCm enablement, router-model training, and open MoM experimentation.

The work has also been carried by FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Chen Wang, Yue Zhu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, and Qiping Pan. Their code, reviews, testing, documentation, and stewardship carried the project from one release to the next.

At this milestone, the project stands at 1,734 commits and 150+ contributors. We thank collaborators at MBZUAI, McGill University, Mila, and Rice University, and the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities. This milestone belongs to everyone who helped turn an early router into a real system.

Figure 14: Building the MoM engine is an open systems problem that needs the full model and infrastructure community.

Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.

vLLM Semantic Router began by helping infrastructure choose the right model for each request.

Now we are extending that foundation beyond a single model: toward systems that can coordinate, evaluate, and operate multiple models across devices and environments.

We invite the community to help build and test that approach in the open.

The Feature Everyone Wants and Almost Nobody Has​

Cursor's Auto mode is deceptively simple: the developer types, and the IDE chooses whether a prompt deserves a frontier model or something faster and cheaper. It is easy to stop noticing — until moving to an open tool where every request starts with a model dropdown.

That gap matters more than it sounds. Teams everywhere are standing up local and internal model serving — a fine-tuned coder on owned GPUs, a frontier API for hard problems, a fast cheap model for everything else. The models exist. The serving works. What is often missing is the decision layer: something that reads each request and sends it to the right backend automatically.

OpenCode ships without a built-in Auto mode, but it does expose an open provider interface — enough to bolt intelligent routing on behind a single OpenAI-compatible endpoint.

This guide walks through building Auto mode for OpenCode (or any OpenAI-compatible client) with vLLM Semantic Router and AgentGateway: one endpoint, one model name, and an ML router choosing among the fleet per request. The configs below are complete; the failure modes are the ones that typically surface first in production setups.

[Demo video: three prompts from one OpenCode session, routed live to three different models]

First, Know Your Options: The Routing Algorithms in vLLM Semantic Router​

"Auto mode" means different things to different teams. vLLM Semantic Router ships more than a dozen selection algorithms, each answering a different operational question:

Decisions can also be gated by signal rules — classifiers for intent, PII, and jailbreak detection. "Anything containing PII stays on the on-prem model" is enforceable as routing policy, not just documentation. When the primary need is governance rather than optimization, signal rules are the right layer.

Practical rule of thumb: interchangeable replicas → multi_factor or latency_aware. Specialist pools (local coder + frontier API) → prompt-aware routing with router_dc. The rest of this guide uses router_dc.

The Design: One Virtual Model Called MoM​

The Semantic Router's core abstraction is a decision: a named bundle of candidate models, a selection algorithm, and a virtual model name the client calls. A typical Auto-mode setup exposes MoM — Mixture of Models:

decisions:
  - name: MoM
    description: "Mixture of Models router"
    priority: 100
    rules:
      operator: AND        # empty AND = catch-all (see gotcha #1 below!)
    modelRefs:
      - model: qwen-coder
      - model: gpt-4o
      - model: gemini-flash
    algorithm:
      type: router_dc

With router_dc, routing rules about prompt keywords are unnecessary. Instead, configure plain-English descriptions of what each model is good at:

modelCards:
  - name: qwen-coder
    description: >
      Specialized coding model optimized for programming tasks.
      Excellent at writing code, debugging, algorithms, ...

  - name: gpt-4o
    description: >
      Frontier reasoning model with exceptional analytical capability.
      Best for complex multi-step reasoning, strategic analysis, ...

  - name: gemini-flash
    description: >
      Fast general-purpose model. Ideal for simple factual questions,
      quick lookups, summarization, casual conversation, ...

At request time, an embedded mmBERT model (CPU, ~130MB) embeds the incoming prompt and compares it to those descriptions by cosine similarity. Example routing logs:

[RouterDC]   qwen-coder:   similarity=0.9998   ← "Write me a Python function..."
[RouterDC]   gpt-4o:       similarity=1.0000   ← "Compare utilitarian and deontological ethics..."
[RouterDC]   gemini-flash: similarity=0.9939   ← "What is the capital of Japan?"

Routing decision cost is typically 1–18ms on CPU. The descriptions are the routing policy — adding a fourth model (a SQL specialist, a legal model) is a new model card, not new application code.

Why this pattern qualifies as true Auto mode:

1. The client stays simple. OpenCode holds no routing logic and no upstream API keys. Swap models, retune descriptions, add candidates — the client config stays fixed.
2. Cost control is structural. In a coding agent, code-shaped traffic lands on the $0 local model; only prompts that genuinely need frontier reasoning incur frontier pricing.
3. It fails soft. With gateway policy failureMode: failOpen, a dead router process lets traffic fall through to a default route. Users see answers, not hard outages.

The Architecture​

The Semantic Router runs as an Envoy ExtProc sidecar to AgentGateway — no extra proxy hop. The gateway pauses each request, streams the body to the router over gRPC, receives a header mutation, and resumes. Routes match on that header:

binds:
- port: 3000
  listeners:
  - routes:
    - matches:
      - headers:
        - name: x-selected-model
          value:
            exact: qwen-coder
      backends:
      - ai:
          provider:
            openAI: {}
          name: ollama
          hostOverride: localhost:11434
    # ... gpt-4o → OpenAI (with backendAuth),
    #     gemini-flash → Gemini (with backendAuth),
    #     plus a failOpen fallback route

A critical security split: the router never holds API keys. It classifies and sets a header; AgentGateway owns backendAuth and injects credentials per upstream. The component making ML decisions on untrusted input should hold zero secrets — especially when the endpoint serves a team rather than a single laptop.

Wiring OpenCode: The Whole Integration Is One Config Block​

OpenCode accepts any OpenAI-compatible endpoint as a custom provider. In ~/.config/opencode/opencode.jsonc:

{
  "provider": {
    "auto_sr": {
      "name": "Auto (Semantic Router)",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:3000/v1"
      },
      "models": {
        "MoM": {
          "name": "MoM",
          "limit": { "context": 32768, "output": 8192 }
        }
      }
    }
  }
}

The Semantic Router intercepts /v1/models and advertises the virtual model, so OpenCode's discovery finds MoM as if it were a real backend. Select the provider, pick MoM, and every prompt is classified server-side before it reaches an upstream LLM.

For observability, AgentGateway ships a built-in UI (build with --features ui, served at :15000/ui) and can persist every request to SQLite with cost attribution:

config:
  modelCatalog:
  - file: base-costs.json
  database:
    url: sqlite://agentgateway.db

Enable request persistence early — it pays off the first time routing behavior needs debugging.

Common Pitfalls​

Four issues show up repeatedly when wiring this stack. Each applies broadly beyond OpenCode.

Pitfall 1: The catch-all rule that matches nothing​

Older examples use rules: {} as "match everything." On current main, an empty rules block evaluates as an OR over zero conditions — which matches nothing. Requests silently fall back to the default model: no errors, plausible answers, zero actual routing. The tell in logs is "decision":"".

The documented catch-all is an empty AND:

rules:
  operator: AND

If the router "works" but never routes, verify the decision matches at all before tuning the algorithm.

Pitfall 2: Choosing an algorithm that ignores the prompt​

multi_factor scores configured quality, pricing, and live latency signals. It is a fleet-balancing algorithm; the prompt never enters the equation. Without quality or pricing data configured, every factor is neutral and traffic sticks on the first candidate indefinitely.

For specialist pools (coder + reasoner + generalist), use router_dc. Reserve multi_factor for interchangeable replicas where cost, latency, and load are the primary tradeoffs.

Pitfall 3: A virtual model inherits its weakest backend's limits​

A typical first failure after routing to a cloud model:

max_tokens is too large: 32000. This model supports at most 16384
completion tokens, whereas you provided 32000.

OpenCode cannot see individual backends behind MoM, so it may request a generous token budget. The virtual model's advertised limits must be the intersection across all candidates — smallest context window, safest output ceiling:

"limit": { "context": 32768, "output": 8192 }

The same intersection rule applies to tool calling, vision, and every other capability flag. Configure for the least capable backend, not the average one.

Pitfall 4: Agentic clients + small local models = malformed tool JSON​

OpenCode is an agent: every request carries a large system prompt and a tool catalog (write, edit, webfetch, ...). Frontier models return structured tool calls. Smaller local models often emit a broken imitation as plain text:

> what is python language

{"name":"write","arguments":{"content":"Python is a high-level,
interpreted programming language...","filePath":"python_language.md"}}

OpenCode replays history on every request, so malformed JSON from earlier turns becomes part of the context and the session self-reinforces the wrong format. Teams running compact local models alongside agentic clients should expect some variant of this behavior.

Diagnosis requires payload capture. AgentGateway can log the full messages array:

frontendPolicies:
  http:
    accessLog:
      database:
        add:
          gen_ai.prompt: llm.prompt
          gen_ai.completion: 'llm.completion.map(c, {"role":"assistant", "content": c})'

A single SQL query against the access log reveals the exact prompt the model received — agent preamble, poisoned history, and all. Debugging LLM pipelines without payload capture is guesswork.

Mitigation: define a custom OpenCode agent whose prompt replaces the built-in agentic system prompt:

"agent": {
  "chat": {
    "description": "Plain chat through the semantic router",
    "mode": "primary",
    "model": "auto_sr/MoM",
    "prompt": "You are a helpful assistant. Answer directly and concisely in clean markdown. Use fenced code blocks for any code. Never output JSON tool calls.",
    "permission": { "edit": "deny", "bash": "deny", "webfetch": "deny" }
  }
}

Also set "tool_call": false on the model entry. Switch to the chat agent in the OpenCode TUI (Tab), start a fresh session (poisoned history does not self-heal), and models in the pool respond in clean markdown. Keep the default build agent for full agentic work with backends that handle tool calls reliably.

Expected Results​

Three prompts, one OpenCode session, one provider, one model name:

Routing overhead stays in the 1–18ms range on CPU. The gateway UI surfaces per-hop tokens and cost; the SQLite log retains the payload trail for post-incident review.

The outcome that matters: no model dropdown, no accidental frontier spend on trivial prompts, and no routing logic in the client — only English model descriptions and an embedding model that routes on semantic fit.

Takeaways​

• Auto mode is a gateway feature, not a client feature. Build it once behind an OpenAI-compatible endpoint and every compatible tool inherits it.
• Match the algorithm to the business need. Specialist models → router_dc. Interchangeable replicas → multi_factor / latency_aware. Maximum savings with escalation → automix. Governance → signal rules (PII, jailbreak).
• A virtual model inherits the weakest backend's limits. Context, output, tool calling — intersection, not average.
• Payload logging is non-negotiable. Silent misroutes and poisoned agent sessions are invisible without the actual bytes on the wire.
• Keep keys out of the router. The component reading untrusted prompts should hold zero secrets; let the gateway own auth.

Teams standing up internal model serving and wanting Cursor-style Auto mode can get there with two YAML files and a JSON block, using AgentGateway and vLLM Semantic Router.

Get Started​

• Install with agentgateway — Kubernetes integration guide
• Router DC selection algorithm — prompt-to-model-description routing
• Giving AgentGateway a Semantic Brain — homelab walkthrough with ExtProc routing
• Ollama local setup — run local models behind the gateway

Published for issue #2424. Have a community story about Semantic Router? Open a blog issue and the team will help get it published.

The more interesting layer may be the one in front of it.

Routers are becoming the control plane for AI inference. Their first role was practical: route the right request to the right model. That already matters because production AI is no longer a one-model world.

A router can cut cost by deciding when a request deserves a frontier model and when an open-source or local model is enough. It can make safety policy executable by sending sensitive domains to stricter models, stricter filters, or stronger review paths. It can coordinate cloud and edge, keeping private or low-latency intent local while escalating harder work to the cloud.

Those are important jobs.

But the next router job is more interesting:

A router can make the model better.

Not by changing weights. Not by asking every application to build a bespoke agent graph. By turning one model API call into a bounded collaboration inside the serving layer.

Figure 1: The router is moving from model selection to capability construction.

This is why Sakana Fugu landed so loudly: it made a commercial product out of a simple but powerful idea, that a "model" can be a surface, and behind that surface can be a team. The research around this idea, including the Fugu technical report and coordination papers such as Conductor and Trinity, gives useful language for thinking about orchestration.

But the vLLM Semantic Router vision is different in where it puts the abstraction. Collaboration should not live only inside one commercial endpoint or one application-specific agent graph. It should become an open serving primitive.

vLLM Semantic Router brings that idea into the open serving layer. The user still calls one model:

{
  "model": "vllm-sr/auto",
  "messages": [{"role": "user", "content": "..."}]
}

Behind that stable model identity, the router can select a recipe, fan out to workers, collect a quorum, verify disagreement, synthesize a final answer, repair the output contract, and return one normal OpenAI-compatible response.

The point is not to expose complexity.

The point is to make collaboration feel like a model.

The Looper Is the Runtime​

In vLLM Semantic Router, the looper is the execution runtime for bounded micro-agents.

A request enters the router as an ordinary chat completion. The router extracts signals, projects them into task-shape or risk bands, matches a decision, and then chooses an algorithm. That algorithm may be a normal single-model route, or it may be a looper route.

Today, the main looper patterns are:

• Confidence: a sequential escalation loop. It tries a cheaper candidate first, measures confidence, and escalates only when the score is too low.
• Ratings: a bounded fan-out loop. It runs multiple candidates under a hard concurrency cap and aggregates them with rating-aware weights.
• ReMoM: repeated mixture-of-model reasoning. It fans out breadth samples, waits for enough successful responses, and runs a final synthesis round.
• Fusion: a panel-judge-final pattern. Independent model responses become evidence for a judge and finalizer.
• Workflows: a micro-agent workflow runtime. It supports static roles or a dynamic planner, executes bounded worker steps, and synthesizes a final response.

Figure 2: Looper algorithms run inside the router while preserving the model API surface.

The implementation details matter. A looper is not a slogan for "ask more models." It is a small runtime with budget, topology, trace, and failure policy.

Confidence: spend escalation only on hard cases​

Confidence is the cost-aware loop. It starts with a smaller or cheaper candidate, then evaluates whether the answer is confident enough to stop. The confidence signal can come from token-level log probability, logprob margin, a hybrid score, self-verification, or an AutoMix-style entailment verifier.

If the score passes the threshold, the router returns immediately. If the score is too low, the route escalates to the next candidate. The important part is not that escalation exists. It is that escalation becomes explicit router policy: thresholds, failure behavior, and stopping conditions are visible and tunable.

Figure 3: Confidence turns escalation into a measured stopping policy.

Ratings: parallel quality under a hard cap​

Ratings is the controlled ensemble loop. It launches several candidates in parallel, but only up to a configured max_concurrent cap. That makes it useful when a route should benefit from multiple model views without turning every request into an unbounded fan-out.

The router collects successful responses, applies rating-aware aggregation, and handles failures according to the route policy. In practice, Ratings is a good fit for A/B-style evaluation, ensemble strategies, and routes where the operator already has meaningful per-candidate quality signals.

Figure 4: Ratings keeps multi-candidate execution bounded and rating-aware.

ReMoM: breadth with a contract​

ReMoM is useful when the task has high reasoning variance and the answer format must survive the collaboration. It fans out multiple reasoning attempts, waits for a minimum-success quorum, then asks a synthesis model to merge evidence into the required output contract.

If synthesis fails but earlier workers produced valid evidence, the route does not have to collapse into an API error. It can fall back to the best valid evidence and still return a normal response.

Figure 5: ReMoM treats breadth, quorum, synthesis, and fallback as serving-time controls.

Fusion: disagreement as signal​

Fusion starts from a different bet. Sometimes the useful object is not the average answer; it is the structure of disagreement. Independent panel answers become evidence. The judge sees agreement, contradiction, and unique insight, then the finalizer returns one answer with the trace collapsed behind the API.

That makes Fusion especially useful when there are plausible competing paths: hard multiple-choice reasoning, long-form expert judgment, or exact-answer tasks where a single confident response can be brittle.

Figure 6: Fusion does not hide disagreement. It turns disagreement into evidence.

Workflows: roles under a budget​

Workflows is the most agentic pattern, and also the one that needs the strictest boundaries. The planner can only choose allowed worker models. The plan is validated. Steps are bounded by max steps, max parallelism, timeouts, and error policy. The final response still has to satisfy the output contract.

For SWE-style tasks, that means the router can express a planner, patcher, verifier, and finalizer without letting the application own a bespoke agent stack. For production serving, that distinction is critical: the loop is powerful, but it is still governed by infrastructure.

Figure 7: Workflows gives the router a bounded role system, not an unbounded autonomous agent.

Auto recipes: one model name, many loops​

The public surface remains one model name: vllm-sr/auto. Internally, the router can use signals and projections to choose the right loop for the request. Difficulty, risk, contract pressure, latency, and cost are not comments in a prompt. They are routing facts that can select Confidence, Ratings, ReMoM, Fusion, Workflows, or a fallback path.

Figure 8: Auto recipes let signals choose the collaboration pattern while preserving one model identity.

This is the difference between "agent as app logic" and "micro-agent as serving runtime." The router controls the budget, policy, topology, trace, and failure mode.

Recipes Beat One Universal Loop​

The most important lesson from our eval work is not that one algorithm always wins.

It is the opposite:

The best loop is task-shaped.

GPQA-Diamond wants strict multiple-choice answer preservation. LiveCodeBench wants runnable code and hidden-test robustness. Humanity's Last Exam wants disagreement resolution and exact-answer formatting. SWE-style tasks need a planner, patcher, verifier, and finalizer.

That is why vllm-sr/auto should not mean "always run the biggest loop." It should mean: select the recipe that fits this task.

Figure 9: Signals and projections let the router choose a benchmark-shaped collaboration pattern.

In our recipes, that shape is explicit:

• GPQA-Diamond routes hard science multiple-choice prompts into a ReMoM recipe with strict ANSWER: X preservation.
• LiveCodeBench looks for constraints, starter code, standard input, float tolerance, timeout risk, and hidden-test risk before selecting a code-shaped loop.
• HLE detects formal reasoning, disagreement risk, long context, and exact answer pressure before choosing between deeper ReMoM, smaller Fusion, or a fallback path.

This is why router-side collaboration is more than prompt engineering. The prompt is only one part. The recipe also defines model pool, model roles, reasoning effort, concurrency, quorum, timeout, synthesis model, fallback policy, output contract, and observability labels.

The Scorecard Is a Proof, Not the Whole Story​

We evaluated the current closed-model recipe across three hard benchmarks. The numbers are useful because they show that the idea is not only aesthetic.

Figure 10: VSR Closed and VSR Hybrid scorecard view across LiveCodeBench, GPQA-Diamond, and Humanity's Last Exam.

In this scorecard, VSR Closed means the recipe uses only closed-model backends. VSR Hybrid means the recipe mixes open and closed models, using the stronger closed models where the recipe needs higher-risk judging, repair, synthesis, or fallback.

The scorecard should be read carefully. It is not a claim that every request should always use every closed model. That would be the wrong product.

The claim is that router-owned collaboration can create a stronger model identity than the individual calls beneath it. It can beat or match frontier single-model baselines while preserving one API surface.

That is the real product shape:

• Users see one model name.
• Operators control the recipe.
• The system can improve without changing the client integration.
• Open and closed models can participate under the same serving abstraction.

What This Means for Model Serving​

The old serving stack was passive. It accepted a model name and sent the request to a backend.

The next serving stack is active. It asks:

• What evidence do we have about this request?
• What quality, cost, latency, and safety band does it fall into?
• Is one model enough?
• If not, what collaboration pattern should run?
• Which answer contract must be preserved?
• What should happen if one provider is slow or wrong?
• How do we expose one clean response while keeping the full trace?

That is not application glue. That is infrastructure.

Micro-agents belong in the router because the router already owns the things micro-agents need: model aliases, provider policy, credentials, cost metadata, signals, decisions, retries, timeouts, traces, and OpenAI-compatible response semantics.

The Takeaway​

The phrase "frontier model" is starting to mean two things.

One is a checkpoint.

The other is a system boundary.

The recent orchestration wave made the direction visible. vLLM Semantic Router is the bet that this capability should be programmable, observable, and open at the serving layer.

The next model race will still involve better models. But it will also involve better routers: routers that know when to save money, when to enforce safety, when to stay on the edge, when to go to the cloud, and when to turn one request into a small, disciplined team.

That is the promise of micro-agents inside the Model API.

Acknowledgements​

We thank researchers from MBZUAI, McGill University, Mila, and Agentic Intelligence Lab, especially Prof. Xue Liu and Dr. Bowei He, for research collaboration and discussions around router-side model collaboration.

Individual Contributors: Huamin Chen, Yincheng Ren.

We also thank AMD's Andy Luo and Haichen Zhang for AMD GPU evaluation support.

Agent systems that span multiple models — a local endpoint for coding, a frontier cloud model for deep reasoning, and a fast general-purpose model for everyday tasks — all face the same routing question: how should each request be directed to the right backend?

Many deployments start with a lightweight Python proxy or keyword matcher in front of the gateway. That approach works at small scale, but misroutes grow quickly as traffic, languages, and task types diversify. This post shows how vLLM Semantic Router running as an Envoy ExtProc sidecar inside AgentGateway replaces that pattern with semantic, config-driven routing.

The Problem: Keyword Routing Does Not Scale​

A typical multi-model agent gateway fronts three backends:

The routing layer was a simple keyword matcher:

# router.py — keyword-based routing
coding_keywords = ["code", "python", "javascript", "bash", "script",
                   "function", "bug", "error", "html", "css"]
reasoning_keywords = ["think", "analyze", "explain in detail",
                      "reasoning", "logic", "deduce"]

if any(k in prompt_lower for k in coding_keywords):
    intent = "coding"
elif len(prompt) > 400 or any(k in prompt_lower for k in reasoning_keywords):
    intent = "reasoning"
else:
    intent = "simple"

After two weeks of sustained traffic, the rough numbers looked like this:

Eighteen percent misroutes is not just wasted spend — it produces worse answers. Scheduled agent jobs that sent "summarize this week's calendar and suggest optimizations" to the 7B local model instead of Gemini or GPT-4o returned noticeably weaker output. Mixed-language prompts and unanticipated domains silently fell through to the wrong lane.

The architecture needed a routing layer that understood the prompt, not just scanned it for keywords.

Enter vLLM Semantic Router + AgentGateway​

AgentGateway maintainers Keith Mattix and John Howard helped shape first-class ExtProc integration with vLLM Semantic Router. The resulting architecture is straightforward.

Instead of a Python reverse proxy sitting in front of the gateway, Semantic Router runs as an Envoy ExtProc sidecar. AgentGateway pauses the request, sends the HTTP body to the SR gRPC endpoint, receives a header mutation (x-selected-model: qwen-coder), and resumes routing. Zero proxy hops. Zero Python processes. Just gRPC-native intelligence inside the gateway's own request lifecycle.

Semantic Router uses an embedded mmBERT model (a 2D Matryoshka embedding model, ~130MB) to classify every prompt and compare it against model descriptions defined in YAML. No keyword lists. No regex. Actual embeddings.

For Kubernetes deployments, the same pattern is documented in the Install with agentgateway guide.

The Setup: Two YAML Files, No Code​

The full integration is defined in two config files.

1. Semantic Router config (config.yaml)​

This tells Semantic Router about the models and how to route between them:

version: v0.3

providers:
  defaults:
    default_model: qwen-coder
  models:
    - name: qwen-coder
      provider_model_id: qwen2.5-coder:7b
      api_format: openai
      backend_refs:
        - name: local-ollama
          endpoint: host.docker.internal:11434
          protocol: http

    - name: gpt-4o
      provider_model_id: gpt-4o
      api_format: openai
      backend_refs:
        - name: openai-cloud
          base_url: https://api.openai.com/v1

    - name: gemini-flash
      provider_model_id: gemini-2.5-flash
      api_format: openai
      backend_refs:
        - name: gemini-cloud
          base_url: https://generativelanguage.googleapis.com/v1beta/openai

routing:
  modelCards:
    - name: qwen-coder
      param_size: 7B
      context_window_size: 32768
      description: >
        Specialized coding model optimized for programming tasks.
        Excellent at writing code, debugging, algorithms, data structures,
        code review, refactoring, and technical implementation in Python,
        Rust, JavaScript, Go. Best for code generation, fixing bugs,
        writing tests, and technical programming Q&A.

    - name: gpt-4o
      param_size: 200B+
      context_window_size: 128000
      description: >
        Frontier reasoning model with exceptional analytical capability.
        Best for complex multi-step reasoning, strategic analysis,
        comparing trade-offs, writing long-form essays, nuanced
        explanations, math proofs, scientific reasoning.

    - name: gemini-flash
      param_size: ~100B
      context_window_size: 1000000
      description: >
        Fast general-purpose model. Ideal for simple factual questions,
        quick lookups, summarization, casual conversation, translations,
        everyday tasks, and when speed matters more than depth.

  decisions:
    - name: MoM
      description: "Mixture of Models router"
      priority: 100
      rules: {}
      modelRefs:
        - model: qwen-coder
        - model: gpt-4o
        - model: gemini-flash
      algorithm:
        type: multi_factor
        multi_factor:
          weights:
            quality: 0.1
            latency: 0.4
            cost: 0.5
          slo:
            max_cost_per_1m: 0.5

The key insight: each model is described in natural language, and Semantic Router uses those descriptions as semantic anchors. When a new prompt arrives, the router embeds it and compares it against these descriptions using cosine similarity. The closest match wins.

2. AgentGateway config (agentgateway_config.yaml)​

This tells AgentGateway to call Semantic Router as an ExtProc sidecar and route based on the header it sets:

policies:
- name:
    name: semantic-router
    namespace: default
  target:
    gateway:
      gatewayName: default
  phase: gateway
  policy:
    extProc:
      host: "127.0.0.1:50051"
      processingOptions:
        requestBodyMode: buffered
        responseBodyMode: none
        requestHeaderMode: send
      failureMode: failOpen

binds:
- port: 3000
  listeners:
  - routes:
    - matches:
      - headers:
        - name: "x-selected-model"
          value:
            exact: "qwen-coder"
      backends:
      - ai:
          provider:
            openAI: {}
          name: ollama
          hostOverride: "localhost:11434"

    - matches:
      - headers:
        - name: "x-selected-model"
          value:
            exact: "gpt-4o"
      backends:
      - ai:
          provider:
            openAI: {}
          name: openai
        policies:
          backendAuth:
            key: $OPENAI_API_KEY

    - matches:
      - headers:
        - name: "x-selected-model"
          value:
            exact: "gemini-flash"
      backends:
      - ai:
          provider:
            gemini: {}
          name: gemini
        policies:
          backendAuth:
            key: $GEMINI_API_KEY

    - backends:
      - ai:
          provider:
            gemini: {}
          name: gemini-default
        policies:
          backendAuth:
            key: $GEMINI_API_KEY

Notice the separation of concerns: Semantic Router never touches API keys. It classifies the prompt and mutates a header. AgentGateway owns downstream auth. That is exactly how production gateways are designed — routing intelligence decoupled from security posture.

The failureMode: failOpen setting means if the SR container crashes or restarts, AgentGateway falls through to the default Gemini route. During SR container restarts, client requests still get answered without interruption.

ARM64 on Apple Silicon: Two Bugs, Two PRs​

On ARM64 hosts (including Apple Silicon), the SR container may start while embeddings remain unavailable:

{
  "msg": "embedding_models_init_completed",
  "embedding_ready": false,
  "tools_ready": false
}

The mmBERT model loaded but the embedding runtime never became ready. Every routing attempt logged:

Failed to embed model qwen-coder: failed to generate batched embedding (status: -1)

Bug #1: Wrong FFI dispatch (#2172)​

The Go router was calling candle_binding.GetEmbeddingBatched() for all model types — but the Rust FFI backend only supports batched embeddings for qwen3 architectures. For mmbert (the default), it returned status: -1.

The fix in PR #2192 adds a dispatch check:

// Only qwen3 supports the batched FFI. Others use single-text FFI.
func candleEmbeddingSupportsBatched(modelType string) bool {
    return modelType == "qwen3"
}

For non-qwen3 models, it gracefully falls back to GetEmbeddingWithModelType(), which works on ARM64.

Bug #2: Missing model files on first boot (#2173)​

On first boot, when the SR container downloaded mmBERT model files from HuggingFace, several required files (like tokenizer.json and config.json) were not being fetched. Fixed in PR #2195.

Both issues were triaged and fixed within days by the vLLM Semantic Router team — particularly @WUKUNTAI-0211 for the FFI dispatch fix and @theohsiung for the file completeness fix. On ARM64/Apple Silicon, pull the latest main and routing works as expected.

Proof: Real Routing Logs​

Send a coding prompt:

curl http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MoM",
    "messages": [
      {"role": "user", "content": "Write me a Python function to compute fibonacci numbers using memoization"}
    ]
  }'

Step 1 — Semantic Router classifies the prompt (1ms):

{
  "msg": "routing_decision",
  "original_model": "MoM",
  "selected_model": "qwen-coder",
  "reason_code": "auto_routing",
  "routing_latency_ms": 1,
  "component": "extproc"
}

Step 2 — AgentGateway routes to Ollama:

info  request
  gateway=default/default
  route=default/route0
  endpoint=localhost:11434
  http.status=200
  gen_ai.request.model=qwen2.5-coder:7b
  gen_ai.response.model=qwen2.5-coder:7b
  gen_ai.usage.input_tokens=41
  gen_ai.usage.output_tokens=366
  duration=22537ms

One millisecond of routing overhead. The rest is upstream generation time.

On container boot, the full pipeline appears in the logs:

{"msg":"embedding_models_init_started","mmbert_configured":true,"use_cpu":true}
{"msg":"embedding_models_initialized","use_batched":false}
{"msg":"selection_factory_initialized","selector_count":14}
{"msg":"startup_complete","embedding_ready":true,"sem_cache_enabled":true,
 "model_selection":true,"extproc_port":50051,"decisions":"MoM"}

Fourteen selection algorithms are registered out of the box — multi-factor, ELO, RL-driven, hybrid, latency-aware, session-aware, KNN, SVM, K-means, and more. The reference configuration uses multi_factor with cost-heavy weighting, but switching algorithms is a single YAML change.

Measured Results​

The cost savings come from fewer misroutes. When "explain the async/await pattern in Rust" correctly goes to local Ollama instead of GPT-4o, that is a $0.003 request instead of $0.03. Across hundreds of daily agent requests, the difference adds up quickly.

Why Agent Builders Should Care​

Whether the deployment runs on a single host or a production fleet in Kubernetes, agents benefit from a routing layer that understands prompts:

1. Cost control is the primary agent problem. Agents generate a lot of requests. Without intelligent routing, every request goes to the most expensive model. Semantic Router's multi_factor algorithm explicitly weighs cost, latency, and quality.

2. Keyword routing does not scale. The moment an agent handles a domain that was not anticipated, keyword-based routing silently fails.

3. AgentGateway + Semantic Router is production-grade. AgentGateway is a Gateway API data plane built in Rust. Semantic Router is an Envoy ExtProc server written in Go and Rust, backed by the vLLM project. This is the same architecture used in Kubernetes clusters with dozens of models.

4. Zero code maintenance. Once model descriptions are written, the routing config stays stable. Semantic Router learns from the descriptions, not from rules that require ongoing updates.

What's Next​

Common follow-on work for this architecture includes:

• Observability — wiring Jaeger and Prometheus to trace every request from client → AgentGateway → Semantic Router → upstream LLM. AgentGateway already emits OpenTelemetry-compatible spans.
• More models — adding specialized models (medical, legal) with just a new model card in YAML. Semantic Router figures out when to use them.

AgentGateway plus vLLM Semantic Router turns a keyword-based proxy into an ML-powered routing plane — with the same ExtProc pattern usable from a single-node setup to a multi-model Kubernetes deployment.

Get Started​

• Install with agentgateway — Kubernetes integration guide
• Gateway API Inference Extension with agentgateway — route to InferencePools with header-based selection
• Multi-factor selection algorithm — cost, latency, and quality weighting
• Community write-up on DEV — extended narrative by Anup Sharma

Have a community story about Semantic Router? Open an issue like #2257 and the team will help get it published.

A coding agent can begin with architecture work, call tools, receive short tool outputs, continue with "fix that", then ask a privacy-sensitive question in the same user session. The latest message may look simple, but the route cannot be chosen from the latest message alone. The router also has to know whether this is a safe moment to switch models.

This guide shows how to deploy that pattern on AMD ROCm with vLLM Semantic Router. You will start one ROCm vLLM backend, serve the agentic routing recipe, open the dashboard, validate the OpenAI-compatible API, and use Inferoa to experience route decisions and Router Learning behavior from an agent client.

Agentic routing is not only choosing a model. It is choosing when to keep one.

The reference recipe is deploy/recipes/agentic-saars.yaml. It is designed to be runnable on a single MI300X-class AMD ROCm host while exercising the same routing shape you would use with a heterogeneous fleet.

What You Will Deploy​

This guide follows the same deployment style as the earlier AMD Developer Cloud walkthrough: start the ROCm backend first, then run vLLM Semantic Router in front of it.

The fastest remote development setup is to expose the API and dashboard ports directly from the AMD host. If those ports are not open, use an SSH tunnel from your local machine:

ssh -L 8899:localhost:8899 -L 8700:localhost:8700 root@<host>

Then use http://localhost:8899/v1 and http://localhost:8700 locally.

What This Recipe Expresses​

The recipe has four route families plus a local fallback:

In the reference deployment, those paths are represented by logical model names such as qwen/qwen3.6-rocm, google/gemini-2.5-flash-lite, google/gemini-3.1-pro, openai/gpt5.4, and anthropic/claude-opus-4.6. For a single-card AMD guide, they can all point at the same vLLM backend. That is intentional: it lets you validate routing policy, headers, replay, and learning behavior without needing five physical model deployments.

The routing behavior still matters. If a request contains an API key, the privacy decision should route locally because the content is sensitive, not because the user remembered to write "keep this local." If a request moves from simple math to hard code review, the selected decision should change. If a tool loop is active, learning should be able to pin the current model even when the base semantic route would otherwise drift.

Why Router Learning Sits After Routing​

Semantic routing answers:

What route best matches this request?

Router Learning answers a second question:

Given the current session or conversation, should we accept that route change right now?

That separation is the key design point. The base decision remains semantic: simple, complex, privacy, and domain signals still choose candidate routes. Router Learning then applies operational memory: previous model, active tool loop, session or conversation identity, idle timeout, switch history, and prefix cache evidence.

Router Learning adjusts the selected route with continuity, reset, cache, and trace policy.

The compact response header surface is method-keyed so more adaptations can be added later:

x-vsr-learning-methods: adaptation,protection
x-vsr-learning-actions: adaptation=keep_base,protection=hold_current
x-vsr-learning-scopes: protection=conversation
x-vsr-learning-reasons: adaptation=base_best,protection=tool_or_protocol_state

Use those headers for live client display. Use x-vsr-replay-id for the full trace: base selected model, final selected model, matched decision, cache evidence, and the stay-vs-switch reasoning.

Conversation Protect vs Session Protect​

The recipe defaults to conversation-level protection:

global:
  router:
    learning:
      enabled: true
      adaptation:
        enabled: true
        strategy: routing_sampling
        candidate_set: decision
      protection:
        enabled: true
        scope: conversation
        identity:
          headers:
            session: x-session-id
            conversation: x-conversation-id

Use scope: conversation when one agent run should stay stable, but a later run in the same application session should be allowed to route again. In this guide, x-conversation-id maps to an agent run, while x-session-id maps to the broader user or client session.

For stricter products, change only the scope:

global:
  router:
    learning:
      protection:
        scope: session

With scope: session, the first selected model can be protected across multiple conversations until the idle timeout resets the session or a decision explicitly bypasses learning.

Conversation protect keeps one run stable. Session protect can hold a model across runs.

Privacy and security routes should normally bypass learning so policy remains hard:

routing:
  decisions:
    - name: local_privacy_policy
      modelRefs:
        - model: qwen/qwen3.6-rocm
      adaptations:
        mode: bypass

That gives the router a clean rule: learning can protect continuity, but it does not soften a privacy boundary.

Prepare The AMD Host​

Before starting containers, make sure the AMD host has:

• an AMD ROCm-capable GPU instance, such as a single MI300X-class machine
• Docker installed and the daemon running
• access to /dev/kfd and /dev/dri
• enough disk for the model cache
• a Hugging Face token if your environment needs one for model download
• ports 8090, 8899, and 8700 reachable, or an SSH tunnel for them

Install vLLM Semantic Router the same way as the AMD Developer Cloud guide:

python3.12 -m venv vsr
source vsr/bin/activate
curl -fsSL https://vllm-sr.ai/install.sh | bash

vllm-sr --help

Create the shared Docker network used by the recipe. The router container will reach the backend by the Docker name vllm:

sudo docker network create vllm-sr-network 2>/dev/null || true

Start vLLM on AMD ROCm​

Use the official vLLM ROCm OpenAI-compatible image. The current vLLM Docker documentation lists vllm/vllm-openai-rocm as the ROCm serving image.

sudo docker run -d \
  --name vllm \
  --network=vllm-sr-network \
  --restart unless-stopped \
  -p "${VLLM_PORT_QWEN36:-8090}:8000" \
  -v "${VLLM_HF_CACHE:-/mnt/data/huggingface-cache}:/root/.cache/huggingface" \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --shm-size 32G \
  -e HF_TOKEN="${HF_TOKEN:-}" \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_USE_AITER_UNIFIED_ATTENTION=1 \
  -e VLLM_ROCM_USE_AITER_MHA=0 \
  --entrypoint python3 \
  vllm/vllm-openai-rocm:latest \
  -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 \
    --port 8000 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --served-model-name qwen/qwen3.6-rocm google/gemini-2.5-flash-lite google/gemini-3.1-pro openai/gpt5.4 anthropic/claude-opus-4.6 \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --max-model-len 262144 \
    --language-model-only \
    --max-num-seqs 32 \
    --enable-prefix-caching \
    --enable-prompt-tokens-details \
    --prefix-caching-hash-algo sha256 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.90

The multiple --served-model-name aliases let one ROCm backend simulate a model portfolio. In a production fleet, those aliases can point at different local vLLM backends, private endpoints, or provider gateways.

Check the backend before starting the router:

curl -s "http://localhost:${VLLM_PORT_QWEN36:-8090}/v1/models" | jq .

sudo docker logs vllm 2>&1 | \
  grep -Ei 'prefix cache|kv cache|maximum concurrency|iteration' | tail -40

If your vLLM build does not populate usage.prompt_tokens_details.cached_tokens for OpenAI-compatible responses, keep --enable-prefix-caching enabled and validate cache behavior through /metrics instead:

curl -s "http://localhost:${VLLM_PORT_QWEN36:-8090}/metrics" \
  | grep -E 'prefix_cache|prompt_tokens_cached|kv_cache' \
  | head

Start vLLM Semantic Router With The Recipe​

Download the published recipe and serve it with the installed CLI:

curl -L -o agentic-saars.yaml \
  https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/recipes/agentic-saars.yaml

vllm-sr serve \
  --platform amd \
  --config agentic-saars.yaml

The recipe exposes the OpenAI-compatible router through Envoy:

http://<host>:8899/v1

The dashboard is available at:

http://<host>:8700

The vLLM backend in the example is reachable at:

http://<host>:8090

Confirm that the router is ready:

curl -s http://<host>:8899/v1/models | jq .

Open the dashboard at http://<host>:8700. If you are using the dashboard onboarding flow instead of the CLI command above, import the recipe from:

https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/recipes/agentic-saars.yaml

That gives you the same routing policy in the dashboard playground, replay viewer, and topology view.

Smoke Test The Route Families​

Every request should send both identity headers:

curl -s http://<host>:8899/v1/chat/completions \
  -D /tmp/agentic-route.headers \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo-session' \
  -H 'x-conversation-id: demo-run-simple' \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "What is 17 * 23? Answer with only the number."}
    ]
  }'

grep -i '^x-vsr-' /tmp/agentic-route.headers

Expected shape:

x-vsr-selected-model: qwen/qwen3.6-rocm
x-vsr-selected-decision: simple_math_fast_path
x-vsr-learning-actions: protection=allow_switch
x-vsr-learning-scopes: protection=conversation

Try a privacy request:

curl -s http://<host>:8899/v1/chat/completions \
  -D /tmp/agentic-privacy.headers \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo-session' \
  -H 'x-conversation-id: demo-run-private' \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "My API key is sk_live_123456. Is it safe to paste this into a public issue?"}
    ]
  }'

grep -i '^x-vsr-' /tmp/agentic-privacy.headers

Expected shape:

x-vsr-selected-model: qwen/qwen3.6-rocm
x-vsr-selected-decision: local_privacy_policy
x-vsr-learning-actions: protection=bypass

For domain routes, use prompts that naturally carry the domain signal, such as legal/health analysis for domain_legal_health, architecture or difficult code work for domain_code_complex, and research synthesis for domain_stem_research.

Try It From Inferoa​

Inferoa is an inference-native agent harness for long-running coding and tool loops. It has native integration with vLLM Semantic Router: when INFEROA_MODE=auto is enabled, the TUI reads vLLM-SR response headers and shows the selected model, selected decision, Router Learning action, cache evidence, and token pressure directly in the agent session. The docs are available at inferoa.agentic-in.ai.

Install the current dev build:

npm install -g inferoa@dev
inferoa --help

Point Inferoa at the router endpoint. INFEROA_MODE=auto tells Inferoa to send requests through vLLM Semantic Router and preserve the route metadata it sees in response headers:

export INFEROA_BASE_URL=http://<host>:8899/v1
export INFEROA_MODEL=auto
export INFEROA_MODE=auto

Start a session:

inferoa

Ask the same route-family prompts you used in the smoke tests. The footer shows the active endpoint mode, selected model, selected decision, and Router Learning state. For example, privacy traffic should show the local model and a learning bypass, while tool-heavy follow-ups can show a hard lock or stay action when Router Learning protects the current run.

Inside the TUI, open tokenmaxxing views for details:

/tokenmaxxing
/tokenmaxxing signals

/tokenmaxxing summarizes turns, selected models, model changes, cache status, tool-loop state, RTK savings, and latency. /tokenmaxxing signals shows the lower-level route and learning evidence for each turn, including action, reason, scope, selected model, selected decision, and replay id when present.

Auto routing should be inspectable: selected model, decision, learning action, and replay id.

Read Cost and Cache Results Carefully​

Cost savings come from two places:

1. The base route avoids sending every request to the strongest path.
2. Router Learning avoids unnecessary switches that would discard continuity or prefix locality.

Earlier Router Learning validation for the same protection mechanism reported a 78.71% estimated physical-model cost reduction across 21,600 deterministic turns, mostly by preventing unsafe or low-value switches. In this AMD guide, the absolute dollar number depends on the logical model prices you configure and whether the backend exposes cached-token evidence. The useful thing to inspect is the direction: simple and privacy routes should avoid expensive lanes, while long warm agent runs should not switch for tiny score differences.

Switching away from a warm long-running agent can be more expensive than the latest short prompt suggests.

Inferoa can surface token pressure, cache evidence, and model changes while you work. Treat those signals as routing telemetry, not a billing statement. Router Replay is the source of truth for why a route stayed, switched, hard-locked, or bypassed learning.

A Minimal Validation Checklist​

Before connecting a local agent to http://<host>:8899/v1, check these:

Validation should prove route intent, learning behavior, replay visibility, and backend cache evidence.

What You Get​

This guide is not a benchmark for one checkpoint. It is a deployment pattern:

• vLLM on AMD ROCm serves the local model backend.
• vLLM Semantic Router turns model: auto into explicit route policy.
• The recipe expresses simple, complex, privacy, and domain decisions.
• Router Learning protects either one conversation or the whole session.
• Privacy and security decisions can bypass learning.
• Inferoa gives an agent-facing view of the selected route, learning action, cache evidence, and token pressure while you work.

The result is not a sticky-session load balancer. It is an agent-aware router: it can say "this request looks simple" and still decide "do not switch right now" when a tool loop, session policy, or prefix-cache checkout says continuity is the better route.

Fusion is the next vLLM Semantic Router primitive for that world. It lets a route run a panel of models, ask a judge model to analyze agreement and gaps, and synthesize one user-facing answer while keeping policy, configuration, and traces inside the router.

OpenRouter's Fusion launch is a useful signal for why this matters now: model panels are becoming a live serving pattern, not just an offline research idea. This post is not about cloning a hosted endpoint. It is about making Fusion a programmable, observable vLLM-SR primitive for Mixture-of-Models serving.

Figure 1: Fusion API turns model diversity into a vLLM-SR routing primitive: panel, judge, synthesis, trace.

The vLLM-SR Thesis​

For years, the default serving question was simple:

Which single model should serve this request?

That question is still useful, but it is no longer enough. Production systems now need policies that can:

• route simple requests to fast low-cost models
• escalate difficult requests to stronger specialists
• preserve session continuity when model switching would hurt context
• apply privacy, safety, and tenant policy before model execution
• fan out to several models when disagreement is valuable
• record the decision path so operators can debug and improve it

This is the core vLLM-SR view: model quality is not only a property of a checkpoint. It is also a property of the serving system around that checkpoint.

The Mixture-of-Models on AMD GPUs work introduced that router-centered view for vLLM-SR: capture signals, select models, coordinate heterogeneous backends, and expose the route. ReMoM extended the same direction into multi-round model collaboration. Fusion adds a more direct panel-judge-synthesis pattern for requests where multiple independent passes are worth the latency.

What Fusion Adds​

Fusion is not the whole Mixture-of-Models story. It is one algorithm in the router's toolbox.

In vLLM-SR, Fusion is part of routing policy rather than a fixed global endpoint:

1. Signals describe the request: domain, complexity, context, safety, feedback, or other evidence.
2. Decisions choose whether this request deserves a normal route or a Fusion route.
3. Fusion-only entry with model: "vllm-sr/fusion" narrows matching to Fusion-capable decisions, so the request still gets intelligent routing without silently falling back to a single-model route.
4. Panel models produce independent candidate answers.
5. A judge model extracts consensus, contradictions, partial coverage, unique insights, and blind spots.
6. A synthesis call returns one user-facing answer.
7. The trace records which models participated and what happened.

That last point is important. A hosted model slug hides most of this. vLLM-SR makes the panel, judge, policy, and trace explicit, so operators can choose where Fusion belongs instead of paying for it on every request.

Why the OpenRouter Result Is a Useful Signal​

OpenRouter's launch is worth discussing because it gives a public proof point for the same systems idea. On DRACO, a deep research benchmark built around hard open-ended tasks, OpenRouter reported that fused panels outperformed individual models.

These are OpenRouter's numbers, not a vLLM-SR benchmark. We read them as external evidence that model composition deserves to be a first-class serving primitive:

The most interesting row for vLLM-SR is the budget panel. It suggests that independent model diversity can recover quality that a single cheaper model lacks. That is exactly the kind of tradeoff a router should control.

How Fusion Works in vLLM-SR​

The implementation is designed around one principle: Fusion should be a routing algorithm, not a global model setting.

The global runtime config only registers which model slugs should trigger direct Fusion execution. The actual panel, judge, error policy, templates, and runtime knobs live on the matched routing decision, because those choices are workload-specific. A research route may want three diverse providers. A code-review route may want two local specialists and one stronger synthesis model. A privacy-sensitive route may keep the whole panel on self-hosted vLLM backends.

Figure 2: Fusion is signal-driven in vLLM-SR. Auto routing can choose any decision; direct Fusion routing chooses among Fusion decisions only; request plugins override execution, not global policy.

vLLM-SR supports three ways to enter the same algorithm:

Once a request reaches the Fusion looper, execution is explicit and observable:

1. Resolve policy. vLLM-SR merges decision-level Fusion config, decision model refs, and request-level plugin overrides.
2. Protect the router. Registered Fusion slugs cannot be used as judge or panel models, so a Fusion request cannot recursively call Fusion.
3. Run the panel. Analysis models execute concurrently, bounded by max_concurrent.
4. Handle failures by policy. on_error: skip allows partial panels; on_error: fail makes provider failure visible immediately.
5. Analyze disagreement. The judge model produces structured analysis over consensus, contradictions, partial coverage, unique insights, and blind spots.
6. Synthesize or call a tool. The final judge/synthesis call returns one assistant response, or an OpenAI-compatible tool_calls response when the client supplied tools.
7. Return trace and accounting. The response can include Fusion trace data, intermediate panel outputs, failed model records, and aggregated token usage across panel, judge, and synthesis calls.

That last item is part of the router value. A caller receives an OpenAI-compatible response, while the operator still gets the system-level view: which decision fired, which models participated, how many iterations ran, what failed, and how much token usage the whole multi-model execution consumed.

This release focuses on the serving primitive: policy-controlled panels, explicit stage contracts, provider interoperability, and traceable execution. The quality question deserves its own larger public eval comparing Fusion, single-model baselines, and frontier panels across shared tasks.

Fusion Is a Decision, Not a Default​

Fusion is useful because some requests benefit from independent model perspectives. It is expensive because it adds panel calls, judge analysis, synthesis, and usually more latency. The production question is not only "can we fuse models?" It is "when is Fusion worth it?"

That is where vLLM-SR matters. model: "vllm-sr/auto" lets the router decide whether a request should use Fusion at all. Simple prompts can stay on a fast single-model route. Hard research, ambiguous analysis, high-stakes synthesis, or tasks where disagreement is valuable can match a Fusion decision. The same signal-decision layer can also encode domain, tenant, privacy, cost, session, or safety policy before the router pays the latency cost.

model: "vllm-sr/fusion" is the explicit path for clients that want Fusion-only routing. It still uses vLLM-SR signals and decisions, but narrows matching to Fusion-capable decisions so it does not silently fall back to an ordinary single-model route. Request-level Fusion plugins are the override path for clients that need to supply a panel for one call.

Figure 3: Fusion is a decision, not a default. vLLM-SR uses policy to decide when the extra latency is worth it.

That gives operators a more useful control plane than a single hosted Fusion slug:

After the Decision: Traceable Fusion​

Once a request reaches a Fusion decision, vLLM-SR runs a small multi-model workflow with explicit stage boundaries. The panel stage returns independent candidate answers. The judge stage turns those candidates into structured analysis. The final stage consumes that analysis to produce one assistant answer, or a tool call when the client provided tools.

The stage contract keeps the system inspectable. If a panel model fails, on_error: skip can continue with partial evidence while recording the failed model, or on_error: fail can stop immediately. If the structured judge output cannot be parsed, vLLM-SR preserves the raw analysis and marks the parse failure instead of hiding it. The final response can include the Fusion trace, intermediate panel outputs, failed-model records, and total token usage across the whole run.

Figure 4: Fusion uses explicit stage contracts so panel output, judge analysis, synthesis, and trace accounting stay inspectable.

This is how Fusion becomes more than a feature. It becomes one implementation of a programmable Mixture-of-Models control plane.

Try It with vLLM-SR​

Let the Router Decide​

Use vllm-sr/auto when you want the router to choose among all configured decisions:

{
  "model": "vllm-sr/auto",
  "messages": [
    {
      "role": "user",
      "content": "What are the strongest arguments for and against carbon taxes?"
    }
  ]
}

If the matched decision uses algorithm.type: fusion, the request enters Fusion. If the matched decision is a normal route, vLLM-SR uses the normal selected model path.

Request Fusion Explicitly​

Use vllm-sr/fusion when the client explicitly wants Fusion-only routing. This still runs signal extraction, but only Fusion-capable decisions are eligible:

{
  "model": "vllm-sr/fusion",
  "messages": [
    {
      "role": "user",
      "content": "What are the strongest arguments for and against carbon taxes?"
    }
  ]
}

Override the Panel for One Request​

The request can also customize the panel. This override is request-scoped; it does not move judge or panel defaults into global config:

{
  "model": "vllm-sr/fusion",
  "messages": [{ "role": "user", "content": "..." }],
  "plugins": [{
    "id": "fusion",
    "model": "google/gemini-3-flash-preview",
    "analysis_models": [
      "google/gemini-3-flash-preview",
      "moonshotai/kimi-k2.6",
      "deepseek/deepseek-v4-pro"
    ]
  }]
}

Use Fusion in Agent Loops​

For agentic applications, keep using the same OpenAI-compatible tool loop. Fusion gives tool-call authority only to the final judge. Panel models and the structured judge-analysis call run text-only: they see the conversation history, including prior tool results, but they do not receive tools or tool_choice.

{
  "model": "vllm-sr/fusion",
  "messages": [
    {
      "role": "user",
      "content": "Find the latest benchmark result and explain whether it changes our launch plan."
    }
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "web_search",
      "parameters": {
        "type": "object",
        "properties": {
          "query": { "type": "string" }
        },
        "required": ["query"]
      }
    }
  }],
  "tool_choice": "auto"
}

In that request, the panel produces independent text analysis, the judge compares the panel, and only the final judge can answer directly or return standard OpenAI-compatible tool_calls. Non-streaming clients receive the regular Chat Completions JSON shape; streaming clients receive tool-call SSE chunks with finish_reason: "tool_calls". Tool results appended by the client are preserved in the next Fusion turn, so multi-round agent loops continue to work.

Configure Entrypoints and Decisions​

The global config registers API entry aliases only:

global:
  router:
    auto_model_names:
      - vllm-sr/auto
      - auto
      - MoM

Fusion slugs are registered under the looper integration:

global:
  integrations:
    looper:
      fusion:
        model_names:
          - vllm-sr/fusion

The per-decision config owns the route semantics, judge, panel, and runtime knobs:

routing:
  decisions:
    - name: deep-research-fusion
      description: Use model diversity for research prompts with high synthesis risk.
      rules:
        operator: AND
        conditions:
          - type: domain
            name: research
          - type: complexity
            name: needs_reasoning:hard
      algorithm:
        type: fusion
        fusion:
          model: google/gemini-3-flash-preview
          analysis_models:
            - google/gemini-3-flash-preview
            - moonshotai/kimi-k2.6
            - deepseek/deepseek-v4-pro
          max_concurrent: 3
          on_error: skip

That separation is deliberate. global is route-independent runtime state. The judge, panel, optional token budget, concurrency, and route semantics belong to the decision.

Operators can opt in to the OpenRouter-style alias when they want compatibility with existing clients:

global:
  integrations:
    looper:
      fusion:
        model_names:
          - vllm-sr/fusion
          - openrouter/fusion

By default, vLLM-SR registers only vllm-sr/fusion.

What Comes Next​

OpenRouter's DRACO result is a strong signal that model panels deserve serious evaluation. Our next step is to make that kind of evaluation reproducible for vLLM-SR and Mixture-of-Models systems:

• run larger public evals beyond smoke coverage
• compare Fusion, ReMoM, AutoMix, Router-R1, and single-model baselines
• study budget panels against frontier-model panels
• expose trace-level diagnostics for disagreement, missing coverage, and judge behavior
• let routing policy decide when the extra latency is justified

The direction is clear. The best answer will not always come from the largest model. Increasingly, it will come from the best model system, and vLLM-SR is where that system should be programmable.

The previous two releases set the stage. Iris made routing decisions composable. Athena rebuilt the model foundation and expanded the router into memory, safety, model selection, long-context signal handling, OpenClaw orchestration, and AMD ROCm deployment. Themis takes the next step: it makes those capabilities easier to operate, easier to inspect, and harder to misuse.

Since v0.2.0, the project has added more than 350 commits across router core, CLI, dashboard, DSL, Kubernetes, protocol compatibility, model selection, safety, replay, and release readiness. The largest value in v0.3 is not a single feature. It is the convergence of those pieces into one stable contract:

signals become projections, projections feed decisions, decisions choose algorithms, and algorithms select models.

That contract now shows up consistently in the router, the CLI, the dashboard, the DSL, the Helm chart, and the operator-oriented deployment surfaces.

Figure 1: Themis turns signals, policy, operators, and model backends into one inspectable routing control plane.

Why Themis?​

Themis represents order, rules, and judgment. That is the right symbol for this release.

Semantic routing is only useful in production if operators can answer basic questions:

• Which signals fired?
• Which decision matched?
• Which model-selection algorithm ran?
• Which model was selected?
• Which safety or replay plugin changed the path?
• Which config version produced this behavior?
• Can the same policy be deployed locally, through the dashboard, and in Kubernetes without becoming three different systems?

Themis is about making those answers explicit. v0.3 keeps the ambition of Athena, but puts stronger boundaries around the runtime, the API surface, and the operational workflow.

Figure 2: The release value is not one isolated feature. It is the connection between stable contracts, inspection, operations, serving, long context, and validation.

What's New in v0.3 Themis?​

1. A Canonical v0.3 Configuration Contract​

The most important Themis change is the new canonical config shape:

version: v0.3
listeners: []
providers: {}
routing: {}
global: {}

Before v0.3, users could encounter overlapping layouts across local Docker, dashboard-generated config, Helm values, CRDs, examples, and older docs. Themis makes config.yaml the steady-state file and aligns the system around the same top-level architecture everywhere.

That cleanup also removes vllm-sr init. The new flow is simpler:

• use vllm-sr serve from an empty directory for dashboard-first setup
• author canonical config.yaml directly for YAML-first workflows
• migrate older files with vllm-sr config migrate --config old-config.yaml
• import supported provider inventories with vllm-sr config import

This is a breaking change, but it is the right kind of breaking change for a pre-1.0 router: fewer config dialects, clearer ownership, and a more durable public contract.

The config path is also stricter at the edges. v0.3 warns on unknown YAML fields, keeps canonical config loading covered by tests, aligns Python CLI models with modern Pydantic configuration, and gates classifier assets more explicitly. The goal is simple: typos and stale config shapes should be caught before they become silent routing drift.

Figure 3: Local YAML, CLI, dashboard, and Kubernetes now converge on the same canonical v0.3 config shape.

2. Signal, Projection, Decision, Algorithm, Model​

Themis makes the router's mental model more explicit:

This matters because v0.3 adds enough routing intelligence that implicit behavior is no longer acceptable. The router now has richer signal families, projection traces, advanced model-selection algorithms, and response-side plugins. Themis keeps those surfaces programmable without turning routing policy into hidden application code.

The current signal catalog is broad enough to describe not only the latest user prompt, but also safety posture, tool loops, user roles, multimodal intent, conversation shape, structured events, and replayable knowledge-base evidence:

Projection outputs are referenced with type: projection, but they are derived routing surfaces rather than another raw signal family. That distinction matters: signals extract evidence, while projections turn evidence into named policy bands such as support_fast, support_balanced, or support_escalated.

The main v0.3 additions are not just more signal names. The release makes signals composable: conversation signals can detect agentic request shape; event signals can route operational payloads; embedding rules can query non-text modalities; and projection outputs can turn noisy evidence into policy-ready bands.

The dashboard topology view, the DSL editor, the compiler/decompiler, and runtime metrics were updated to understand these v0.3 surfaces instead of silently dropping or hiding them.

The policy-authoring surface is also stronger. The routing DSL gained conflict detection, SIGNAL_GROUP, TEST, and TIER authoring constructs, a natural-language-to-DSL pipeline, EMIT retention, and dynamic tool retrieval support. That matters for production teams because Themis policies are not just parsed YAML; they are reviewable routing programs with tests, retained outputs, and safer generation paths.

Figure 4: The routing contract is now visible as a pipeline from request evidence to signal, projection, decision, algorithm, model, and replay.

3. Session-Aware Agentic Routing​

Themis includes the first production-ready version of Session-Aware Agentic Routing (SAAR).

Single-turn routing asks:

Which model should handle this prompt?

Agentic routing also has to ask:

Is it safe to switch models inside this session right now?

SAAR adds router-owned session memory, hard locks around tool loops, provider-state portability checks, idle and decision-drift reset boundaries, switch economics, and replayable diagnostics. It keeps the normal Semantic Router pipeline, but wraps model selection with session continuity rules.

This is especially important for coding agents and long-horizon tool loops. A tool result should usually return to the model that asked for the tool. A provider-managed continuation id should not be sent to a different physical backend. A long warm session should not throw away prefix locality just because the latest user message is short.

Themis makes those constraints part of the model-selection policy instead of asking every application to rediscover them.

Figure 5: SAAR keeps multi-turn agent sessions stable by combining router-owned session memory, hard locks, portability checks, switch economics, and replay diagnostics.

The key design choice is that SAAR does not replace semantic routing. It adds a stateful guard around the last mile of model selection:

• conversation signals identify multi-turn shape, active tool use, developer messages, and heavy non-user context.
• session_aware selection evaluates whether a model switch is worth it after considering quality gap, switch margin, stay bias, prefix locality, and remaining-turn priors.
• Hard locks stop unsafe switches during active tool loops or provider-state continuations.
• Router-owned memory can retrieve and store route-local facts, preferences, and context without exposing a separate session-state DSL.
• Replay records preserve the reason a session stayed, switched, or reset.

Router memory is the durable complement to session-aware selection. The memory plugin can preserve facts, preferences, and retrieved context under user or session scope; session_aware can then avoid treating every turn as an isolated request. In practice, that means an agent can keep useful continuity without pinning every request to the most expensive model forever.

The reference policy shape is intentionally ordinary YAML:

routing:
  signals:
    conversation:
      - name: active_tool_use
        feature:
          type: count
          source:
            type: assistant_tool_cycle
        predicate:
          gte: 1

  decisions:
    - name: agentic_session_route
      rules:
        operator: AND
        conditions:
          - type: conversation
            name: active_tool_use
      algorithm:
        type: session_aware
        session_aware:
          base_method: hybrid
          tool_loop_hard_lock: true
          context_portability_hard_lock: true
          prefix_cache_weight: 0.20
          handoff_penalty_weight: 1.0
      plugins:
        - type: memory
          configuration:
            enabled: true
            retrieval_limit: 6
            auto_store: true
            hybrid_search: true

That is the part of Themis that matters most for agentic workloads: the router can now reason about continuity, not only classification.

4. Projections Turn Evidence Into Policy​

Signals are raw evidence. Projections are where Themis turns that evidence into named, stable policy concepts.

Without projections, a complex policy has to repeat low-level signal details across many decisions: exact embedding rule names, complexity thresholds, context boundaries, and knowledge-base scores. With projections, the router can compute the raw evidence once, derive a reusable output such as support_fast or support_escalated, and let decisions route on that derived concept.

Themis supports three core projection patterns:

• partitions choose one winner from an exclusive family, such as competing support intents.
• scores combine declared signals or knowledge-base metrics into a continuous value.
• mappings turn those values into policy bands through calibrated thresholds.

For policies that need more than one derived output, v0.3 also adds multi_emit projection mappings. That lets a single projection step emit multiple named routing concepts while still preserving traceability in replay.

Figure 6: Projections transform noisy signal evidence into named outputs that decisions can reference directly.

A compact example looks like this:

routing:
  signals:
    embeddings:
      - name: technical_support
        threshold: 0.75
        aggregation_method: max
        candidates:
          - installation guide
          - troubleshooting steps
      - name: account_management
        threshold: 0.72
        aggregation_method: any
        candidates:
          - password reset
          - billing information
    context:
      - name: long_context
        min_tokens: 32K
        max_tokens: 256K

  projections:
    partitions:
      - name: support_intents
        semantics: exclusive
        members:
          - technical_support
          - account_management
        default: technical_support
    scores:
      - name: request_difficulty
        method: weighted_sum
        inputs:
          - type: embedding
            name: technical_support
            weight: 0.18
            value_source: confidence
          - type: context
            name: long_context
            weight: 0.18
    mappings:
      - name: request_band
        source: request_difficulty
        method: threshold_bands
        outputs:
          - name: support_fast
            lte: 0.20
          - name: support_escalated
            gte: 0.45

  decisions:
    - name: escalated_support_route
      rules:
        operator: AND
        conditions:
          - type: projection
            name: support_escalated

Projection traces are also stored with replay records, so the dashboard can explain not only which signal fired, but also which derived policy band caused the final route.

5. Protocol Compatibility Becomes a Release Surface​

v0.3 expands the router's compatibility boundary beyond basic OpenAI Chat Completions.

The protocol work in this cycle includes:

• native Anthropic /v1/messages ingress through an internal request envelope
• Anthropic streaming with OpenAI SSE translation
• custom Anthropic upstream routing and tool-calling support
• outbound Anthropic response emission for non-streaming paths
• protocol detection from request path headers
• session-id mirroring and header pass-through controls
• response headers that explain when protocol translation is lossy
• Responses API tool-trace fidelity and OpenAI SDK-aligned message handling
• OpenAI reasoning-effort mutation fixes
• identity-encoded upstream responses to avoid transparent decompression surprises
• stronger Responses API state and persistence paths

The goal is not to make every provider look identical. The goal is to make translation explicit, observable, and safe enough that a logical routing model such as auto can sit in front of multiple provider protocols without surprising operators.

6. The Dashboard Becomes an Operator Console​

The Themis dashboard is more than a config editor.

The v0.3 cycle tightens the first-run setup flow, topology graph, replay-backed insights, logs, status pages, evaluation flows, auth behavior, and model inventory surfaces. Operators can import a profile, validate it, activate it, send test prompts, inspect signal paths, read router logs, and verify replay records without leaving the dashboard.

Figure 7: The dashboard becomes a practical operator console for setup, topology inspection, logs, playground testing, replay, and model health.

Notable dashboard improvements include:

• built-in routing modes and missing-model completion
• topology dry-run paths that show matched signals, projections, decisions, and models
• router replay and aggregate insights through the dashboard proxy
• natural-language DSL builder and evaluation-flow fixes
• file attachments in the playground
• auth fail-closed behavior when the auth service cannot initialize
• policy version lifecycle with shadow, activate, and revert states
• safer logs and URL redaction for user-supplied fetch/open-web requests
• UTF-8-safe display handling for multilingual content
• slimmer production route shell and smaller backend runtime dependencies
• dashboard-aware model list and status surfaces

The result is a better local and remote operator workflow: setup mode for first run, topology for policy inspection, logs/status for operations, and insights for real traffic.

7. CLI and Deployment Are More Predictable​

Themis also strengthens vllm-sr as the supported operating interface.

The CLI now has clearer runtime boundaries and more useful commands:

vllm-sr serve
vllm-sr serve --algorithm latency_aware
vllm-sr serve --algorithm session_aware
vllm-sr serve --platform amd
vllm-sr serve --platform nvidia
vllm-sr chat
vllm-sr eval
vllm-sr model list
vllm-sr config migrate --config old-config.yaml

Local vllm-sr serve remains a Docker-based workflow on Linux, macOS, and WSL2. AMD ROCm remains the release-validated GPU path, while --platform nvidia adds local NVIDIA Docker passthrough ergonomics for users who already have the NVIDIA container runtime configured. Native Windows Docker serving is now rejected with an explicit support message rather than failing later in less obvious ways.

The CLI also grows better inspection and smoke-test commands. vllm-sr model list surfaces configured model inventory, vllm-sr chat provides a one-shot completion path, vllm-sr eval exercises router evaluation endpoints, and VLLM_SR_DNS lets local containers join custom DNS environments when enterprise or lab networks require it.

On Kubernetes, v0.3 aligns Helm, release defaults, OpenShift deployment fixes, multiple IntelligentRoute reconcile behavior, CRD modality contracts, optional Gateway API HTTPRoute ingress, and AgentGateway installation guidance. For release operations, Themis also moves away from vague latest assumptions and toward explicit artifact contracts, upgrade and rollback documentation, and release checks.

8. Safety, Replay, Memory, and Retrieval Are More Trustworthy​

Athena brought many of these capabilities into the router. Themis hardens them.

Key runtime fixes and improvements now fall into three groups:

Replay and observability

• router replay PostgreSQL insert correctness so dashboard insights do not silently stay empty
• projection traces stored with replay records for better explainability
• response-side jailbreak and replay path tightening

Storage and retrieval

• Qdrant vector search provider support
• Valkey cache, vector store, and memory backend support, including TLS and search-module prechecks
• Redis and Responses API storage defaults that better match real local and Kubernetes deployments
• hybrid cache rebuild preallocation reduction
• streaming Redis semantic-cache correctness and bounded streaming chunk memory behavior
• O(N) cache-LRU read paths replaced with a constant-time list-backed implementation
• BM25 and n-gram classification caching to avoid amplified work
• hybrid HNSW entry-point propagation fixes
• shared Milvus lifecycle handling across replay, cache, memory, and vector store paths

Runtime and security hardening

• history-aware PII and jailbreak signal scanning across prior user turns
• model switch gate fixes for previous-model population
• goroutine panic recovery in extproc background paths
• concurrency race fixes in selection randomness
• path traversal protection for config rollback versions
• dependency security updates across Python, Go, Rust, and frontend surfaces

This is the less flashy part of the release, but it is exactly what Themis is for: making the system safer under real traffic, long prompts, replay storage, and operator-driven config changes.

9. Long-Context Routing Gets Cheaper​

Themis adds three important long-context controls.

First, context token estimation can now learn an online calibration ratio from observed response usage, so context-sensitive routing can improve when exact tokenization is unavailable. The fallback remains conservative, but the router can adapt to real traffic over time.

Second, the native mmBERT embedding path now bounds memory without turning long inputs into a silent clipping problem. The #2007 native-binding fix for the long-input memory issue processes attention in query chunks instead of materializing one dense attention tensor for the whole sequence. That keeps the long-context signal available to the router while making the binding usable under larger prompts.

Figure 8: The long-context path preserves the signal and bounds native memory by chunking mmBERT attention work.

Third, prompt compression becomes a named profile surface for signal extraction:

The compression path is intentionally scoped to signal evaluation. The original user prompt still goes to the selected serving model unless a decision-owned plugin explicitly changes it. That separation keeps routing optimization from silently rewriting user intent.

10. Hardware Backend Paths Broaden​

Themis broadens the router-owned model execution story beyond the default local path.

The broadened map separates four paths: NVIDIA CUDA and AMD ROCm for served vLLM backends, Intel OpenVINO for router-owned classifier and embedding inference, and CPU/local execution for development and smoke tests.

On Intel infrastructure, v0.3 adds an initial OpenVINO binding for Semantic Router. The new binding provides native C++ and Go integration for ModernBERT sequence classification, token classification, and embedding inference, with benchmark entrypoints that compare OpenVINO and Candle behavior for classifier and embedding workloads.

This is a backend and binding milestone, not a blanket production-parity claim. It gives contributors and hardware partners a concrete path to validate Semantic Router's internal classifier and embedding models on Intel OpenVINO while preserving the same routing contract used by the rest of Themis.

Figure 9: Themis broadens the hardware backend map while keeping one routing control plane across NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and CPU/local paths.

The AMD deployment path introduced in Athena also remains part of the v0.3 release contract.

The reference flow is still:

vllm-sr serve --platform amd

For real AMD deployments, the project keeps the maintained deploy/recipes/balance.yaml profile, which exposes multiple served aliases through a ROCm vLLM backend and routes them through the same signal, projection, decision, and model-selection pipeline as the CPU/local path.

As part of release readiness, Themis was validated on an AMD ROCm stack with:

• a ROCm vLLM backend exposing the expected served aliases
• dashboard setup import, validate, and activate using the reference balance profile
• router health and Envoy OpenAI-compatible /v1/models
• topology dry-run for a coding/debug request
• direct Envoy chat completions for coding, math, and legal prompts
• dashboard proxy chat completions
• router replay list and aggregate insight APIs

Figure 10: The AMD release path validates serve, dashboard import, router health, model listing, ROCm backend serving, and routed requests as one flow.

That end-to-end path is important because Semantic Router is meant to be a control plane across heterogeneous inference stacks, not only a local development tool.

11. RouterArena SOTA Refresh​

Themis also comes with an external leaderboard signal: in the RouterArena snapshot captured for this release update, vLLM-SR returned to #1 on the RouterArena leaderboard.

In that public RouterArena leaderboard snapshot, vLLM-SR is ranked first by weighted Arena Score with a score of 75.4, ahead of Sqwish Router, AgentForge Router, Nadir Router, and other published router baselines. The same snapshot reports 76.0 accuracy, $0.11 cost per 1K queries, and 73.1 robustness for vLLM-SR.

Figure 11: RouterArena leaderboard snapshot showing vLLM-SR back at #1 by weighted Arena Score.

This is not a substitute for release testing, but it is a useful outside check on the project direction. Themis improves routing policy, cost-aware selection, protocol compatibility, and operational traceability while keeping the router competitive on independent router benchmarks.

What Changed Since v0.2?​

At a high level, the v0.2 to v0.3 delta looks like this:

That is the core Themis story: the router is more capable, but also more constrained in the right places.

Get Started​

For macOS or Linux:

curl -fsSL https://vllm-semantic-router.com/install.sh | bash

For manual installation:

pip install vllm-sr==0.3.0
vllm-sr serve

If the current directory does not contain config.yaml, vllm-sr serve starts the dashboard in setup mode. For YAML-first users, create a canonical v0.3 config directly or migrate an older file:

vllm-sr config migrate --config old-config.yaml
vllm-sr serve --config config.yaml

For AMD ROCm:

vllm-sr serve --platform amd

For local NVIDIA Docker passthrough:

vllm-sr serve --platform nvidia

For Kubernetes:

helm install semantic-router oci://ghcr.io/vllm-project/charts/semantic-router

See the project resources:

• Documentation: vllm-semantic-router.com
• GitHub: vllm-project/semantic-router
• Reference AMD profile: deploy/recipes/balance.yaml
• Models: Hugging Face

Looking Ahead: v0.4 Hermes​

The next release codename is Hermes.

Themis makes the contract stable enough to operate. Hermes should make the router faster to improve, easier to evaluate, and safer to adapt under real workloads. The core Hermes goal is a self-improving router. The loop is deliberate: run auto research for router performance at GPU scale, tune DSL recipes with router evaluation, then feed validated evidence back into the codebase and encoder-model fine-tuning. The highest-value work is:

• Self-improving router as the Hermes core goal: close the loop across GPU-scale performance research, DSL recipe tuning, and codebase plus encoder-model fine-tuning. Every generated change still has to be reviewable, replayable, versioned, and rollback-safe.
• SAAR as the agentic routing layer: continue tightening model-switch economics, tool-loop continuity, provider-state portability, replay diagnostics, and router memory integration.
• Evaluation as a release gate: build system-level and signal-level evaluation so every signal, projection, algorithm, plugin, and dashboard path can be replayed against representative traffic before release.
• CLI-first design: make sure every Semantic Router operation can close the loop through vllm-sr, including config authoring, migration, serving, inspection, evaluation, replay, policy lifecycle, dashboard import/export, and release smoke tests.
• Better router-owned models: improve accuracy and latency for the models the router itself uses, including embedding, classifier, multimodal, and safety signal models.
• More useful signals: add richer request, response, tool, modality, identity, freshness, latency, cost, and runtime-health signals without turning the DSL into application code.
• Operator debugging loop: make what-if routing, policy replay, evaluation-driven tuning, and trace comparison first-class dashboard workflows.

Figure 12: Hermes centers on a self-improving router that connects GPU-scale performance research, DSL recipe tuning, router evaluation, codebase updates, and encoder fine-tuning.

Acknowledgments​

From v0.2.0 to v0.3.0, the Themis cycle includes more than 350 commits from 80+ contributor author identities. Thank you to everyone who reviewed code, improved docs, trained models, hardened tests, fixed release blockers, and pushed the router toward a more stable production shape.

We separately thank collaborators from research institutions and universities, including MBZUAI, McGill University, Mila, and Rice University, for contributions and collaboration across router evaluation, model research, and AI systems.

We also thank the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities for continued collaboration across runtime systems, model serving, model research, and production AI infrastructure.

Welcome to Themis: from signals to stateful production routing.

This post introduces Session-Aware Agentic Routing (SAAR), a session-aware model selection policy in vLLM Semantic Router. SAAR keeps semantic routing, but adds router-owned session memory, hard locks around tool loops and non-portable provider state, safe reset boundaries, prefix-cache-aware switch pricing, and replayable traces.

Across 21,600 deterministic turns, SAAR cuts model switches by 79.29%, eliminates 3,836 unsafe switches, and reduces estimated physical-model cost by 78.71%. Across 2,896 live AMD ROCm requests, it preserves session continuity with 0 observed violations.

Figure 1: Long-horizon agents need routing decisions that understand the session trajectory, not only the latest prompt.

From Prompt Routing To Session Routing​

vLLM Semantic Router started from a simple systems observation: not every request should take the same path through an inference stack. A short factual question, a security-sensitive prompt, a multimodal request, a hard reasoning task, and a domain-specific query may all deserve different treatment.

The first generation of that idea was prompt routing. The router extracted signals from the current request, matched a routing decision, and selected an appropriate path. Iris made those signals composable. Athena made the router more strategic by expanding model selection, memory, replay, long-context signals, multimodal primitives, and AMD ROCm deployment paths.

Agents change the unit of routing again.

A coding or research agent is not one prompt. It is a session. It plans, calls tools, receives tool outputs, edits files, runs tests, recovers from errors, pauses, resumes, and often sends very short follow-up messages such as "continue", "fix it", "run that again", or "use the previous result." Those turns are meaningful only because of the trajectory that came before them.

That is why this milestone matters for Semantic Router. The router is no longer answering only:

Which model should handle this request?

For agent traffic, the router also has to answer:

Is it safe to switch models inside this session right now?

That second question is what SAAR is designed to handle.

Why Single-Turn Routing Breaks Down For Agents​

Single-turn routing can be locally correct and still be wrong for the session.

Consider a typical tool-using agent loop:

The latest message alone does not contain enough information. A prompt router may decide that the tool result looks cheap and send it to a smaller model. It may see a generic "continue" and re-run the normal selector. It may miss that provider-managed continuation state belongs to one physical backend. It may discard a warm prefix cache for a frontier model because the current message is short.

Each of those mistakes has a different failure mode:

• A tool result can go to a model that did not make the tool call.
• A non-portable continuation id can be sent to the wrong physical backend.
• A long, warm session can lose prefix locality and become unnecessarily expensive.
• A logical model such as auto can become hard to debug because users no longer know which physical model actually served the turn.

The important point is not that agents should never switch models. They should. A good router should still move from a cheap model to a stronger model when the task becomes harder, and it should move back when the session reaches a safe boundary. The problem is that the router needs session context to know which moments are safe.

The SAAR Design​

SAAR keeps the existing Semantic Router decision pipeline. Signals are still extracted from the request, decisions are still matched, and model-selection algorithms still rank candidate models inside a matched decision.

SAAR adds a session-control layer around that result.

Figure 2: SAAR combines router memory, hard locks, reset boundaries, switch economics, and replayable traces before selecting a physical model.

There are five pieces:

This is a model-selection policy, not an endpoint load balancer. Semantic Router can choose a model or cluster through the gateway contract. Endpoint membership, health checks, and load balancing inside a cluster remain infrastructure responsibilities.

The Most Important Rule: Sometimes The Router Must Not Switch​

The safest model switch is not always the one with the best score on the latest prompt. For agent traffic, some turns are continuity-constrained.

Figure 3: Tool loops and provider-managed continuation state are hard continuity constraints; idle and decision-drift boundaries permit safe reselection.

SAAR treats two cases as hard locks:

• Tool-loop continuity. If a physical model asked for a tool call, the tool result should return to that same physical model. The follow-up observation is not a fresh prompt; it is part of a local execution loop.
• Provider-managed state. If the request carries non-portable continuation state, such as a response identifier that belongs to one backend, SAAR holds the previous physical model instead of silently moving the state elsewhere.

These rules are intentionally stronger than cost rules. If a switch is unsafe, the router should not "buy" its way out with a cheaper model.

SAAR also defines the opposite boundary: when the router may switch again. Idle timeout and decision drift reopen the selection. If an agent pauses long enough, the value of continuity decays. If the matched decision changes because the user moved from code editing to synthesis or from retrieval to debugging, the old model choice should not stick forever.

This distinction is the heart of session-aware agentic routing:

Router Memory Is Not User Memory​

The phrase "router memory" can be misleading, so the boundary is important.

SAAR memory is not conversation memory, retrieval memory, or user profile memory. It does not summarize the conversation and it does not try to remember facts for the model. Its job is narrower: keep enough routing state to make the next model-selection decision safe and explainable.

For each session, the router tracks facts such as:

• the last physical model selected behind the logical model;
• the last matched routing decision;
• whether the session is in a normal, tool-loop, provider-state, idle-reset, or drift-reset phase;
• how many recent switches happened;
• the latest context length and cache evidence;
• a replay id that links the response back to the router's decision trace.

That scope keeps the system operationally useful without turning the router into a second agent memory layer. Application memory should remain in the application. Retrieval memory should remain in the retrieval stack. SAAR memory exists only to make routing across turns coherent.

Prefix Cache Makes Model Switching Asymmetric​

For long agent sessions, model switching is not just a quality decision. It is also an input-side systems decision.

Figure 4: The same switch has a different cost depending on model tier, session length, and physical prefix reuse.

A short retry on a cheap model and a 40-turn warm session on a frontier model should not be treated the same way. The latter has accumulated a valuable prefix. Switching away from it may require the next physical model to pay a much larger input cost even if the visible user message is short.

SAAR therefore prices a cached-input checkout delta: the gap between normal prompt input price and cached-input price for the physical model under consideration. The longer and more expensive the session, the stricter the policy becomes about discarding prefix locality.

This also clarifies cached-token accounting for a routed logical model. If the user calls auto, the router may map that logical name to different physical models over time. A cache hit reported by one backend is physical evidence for that backend. It is not automatically transferable to another backend. SAAR keeps backend-reported cached tokens separate from router-estimated reuse, and it does not rewrite upstream usage fields.

That separation is useful operationally. Operators can still inspect physical cache behavior while the router uses its own memory to decide whether switching is worth the checkout cost.

How A Request Moves Through SAAR​

The serving path stays familiar. Clients send requests to the OpenAI-compatible gateway, usually with a logical model name such as auto. To enable session-aware routing, they also send a stable session identifier such as x-session-id.

SAAR then handles each turn in this order:

1. Read the current request, session id, tool-call context, provider-state markers, and candidate model set.
2. Run the normal Semantic Router signal and decision pipeline.
3. Produce a base model-selection result from the configured method, such as hybrid scoring.
4. Load the previous session routing state from router memory.
5. Apply hard locks for tool loops and provider-managed state.
6. Check idle timeout and decision drift boundaries.
7. Adjust switch scores using prefix-cache checkout cost and switch history.
8. Select the physical model and emit diagnostics.
9. Update router memory and write a replay trace.

The configuration lives inside a routing decision's model-selection algorithm:

routing:
  decisions:
    - name: agentic_routing
      modelRefs:
        - model: qwen3-8b
        - model: qwen3-32b
      algorithm:
        type: session_aware
        session_aware:
          base_method: hybrid
          idle_timeout_seconds: 300
          tool_loop_hard_lock: true
          context_portability_hard_lock: true
          decision_drift_reset: true
          prefix_cache_weight: 0.20
          switch_history_weight: 0.04

The values are intentionally policy knobs, not one-size-fits-all constants. A customer-service assistant with short sessions may use a more permissive idle boundary. A coding agent with long tool loops and expensive context may use stricter continuity and prefix-cache settings.

Observability Is Part Of The Feature​

Model selection behind auto is only useful if operators can explain it.

Figure 5: SAAR turns hidden physical routing choices behind a logical model into inspectable traces and response headers.

SAAR emits diagnostics such as selected model, selected decision, replay id, session phase, selected confidence, and context-token count. The replay id joins a served response back to the router trace that explains the decision.

A useful trace answers questions like:

• What model would the base selector have chosen?
• Did the router hold the previous model because of a tool-loop lock?
• Did provider-managed state make switching unsafe?
• Did the session cross an idle or drift boundary?
• How did prefix-cache evidence change the adjusted candidate scores?
• Was the final decision a stay, a switch, or a locked stay?

This makes session-aware routing operable. Without replay, a router behind a logical model becomes hard to debug. With replay, an operator can audit why the router preserved continuity or decided that a switch was safe.

How We Evaluate It​

The evaluation is designed around one question: does the policy make routing more agent-friendly without hiding correctness problems?

We use three layers of evidence.

First, a deterministic policy matrix tests the control logic across many synthetic sessions. This isolates the routing policy from serving noise and lets us stress tool loops, provider state, idle boundaries, drift boundaries, model tiers, and switch history.

Second, live OpenAI-compatible serving runs exercise the same invariants through the router and backend serving path on AMD ROCm. This checks that headers, session ids, diagnostics, and failure handling survive real request flow.

Third, deterministic agent-task traces add task structure. Instead of only counting switches, these traces include simulated tool observations and exact final-answer scoring.

The goal is not to make every plot say "fewer switches." Sticky sessions can do that. The goal is to show that SAAR removes unsafe switches, keeps useful movement, respects expensive prefix locality, and remains observable in live serving.

Result 1: SAAR Moves The Unit Of Control From Turn To Session​

The deterministic policy matrix covers balanced, tool-heavy, frontier-heavy, idle-heavy, provider-state-heavy, and drift-heavy sessions. Each workload runs five seeds, 40 sessions per seed, and 18 turns per session, for 21,600 total turns.

Figure 6: Headline policy result across 21,600 deterministic turns.

The headline result is that SAAR reduces model churn while preserving the ability to move:

Single-turn routing switches often and creates unsafe movement. Sticky sessions nearly eliminate movement, but they also give up too much quality because they refuse to reselect after the task changes. Full SAAR sits in the middle for the right reason: it removes unsafe movement while still letting idle and drift boundaries reopen the decision.

This is the shift from turn-level control to session-level control. The router is no longer treating every message as a fresh independent event.

Result 2: Hard Locks Remove The Correctness Failures​

The second result isolates the most important invariant: when switching is unsafe, SAAR should not switch.

Figure 7: Hard locks remove unsafe switching during tool loops and non-portable provider state.

Tool-loop switch violations fall from 3,404 to 0. Provider-state switch violations fall from 432 to 0.

These are not minor tuning wins. They are correctness boundaries. A tool result is not an ordinary prompt. A non-portable continuation id is not an ordinary text field. If a router ignores those facts, it can break the interaction while still appearing to make a reasonable semantic choice on the latest message.

SAAR fixes that by making continuity constraints explicit in the policy.

Result 3: SAAR Is Not Sticky Sessions With A New Name​

The obvious baseline is sticky routing: pick the first model for a session and hold it.

Sticky routing is attractive because it is easy to reason about. It also solves many unsafe switch cases by avoiding switching entirely. But that simplicity becomes a product problem for agents. Long sessions drift. Users change tasks. A cheap initial model may no longer be appropriate. A strong model may no longer be necessary.

Figure 8: SAAR is not just sticky sessions; it balances continuity with movement.

The ablation shows why SAAR needs multiple mechanisms:

Locks provide correctness. Reset boundaries provide liveness. Prefix-cache checkout pricing provides economic discipline. Removing any one of them changes the behavior in a way that is visible in the metrics.

Result 4: The Invariants Hold In Live AMD ROCm Serving​

Policy simulation is useful, but a router has to work through real request flow. The live serving runs use OpenAI-compatible traffic through the router and AMD ROCm backend paths, with matched schedules for routed and direct-backend runs.

Figure 9: Live ROCm runs preserve continuity under long sessions and injected backend failures.

Across long-session runs, the router completes 2,896 live requests with 0 observed continuity violations.

The idle workload includes real wall-clock sleeps, so its p95 overhead should be interpreted separately from hot-path routing overhead. The important result is continuity: the live path preserves the hard-lock and reset-boundary behavior tested in the deterministic matrix.

Result 5: Sessions Recover After Backend Faults​

Long-horizon agents need session-level recovery, not just per-request success. A backend can return HTTP 503 for one request, but the session should continue later without losing the routing invariants that keep the interaction coherent.

Across the one-shot disruption matrix, 32/32 affected sessions recovered later. Across the repeated-failure matrix, 24/24 affected sessions recovered after 168 injected HTTP 503 responses.

This matters because agent sessions are longer than ordinary chat turns. A transient backend fault should not make the router forget that a tool loop is active, that provider state is non-portable, or that the session has a replayable history.

Result 6: Task Traces Exercise The Agent Loop​

Continuity counters are necessary, but they are not enough. We also run deterministic multi-turn task traces with simulated tool observations and exact final-answer scoring. There is no judge model in the loop: the final answer either contains the required labels or it does not.

In the AMD serving task run, 18/18 exact-scored task instances complete, replay headers are present on 96/96 routed turns, and no continuity violation is observed.

This is still smaller than a broad real coding-agent benchmark, but it is a stronger signal than policy counters alone because it exercises a task loop with tool observations and final-answer checks.

What This Changes For vLLM Users​

Session-aware routing makes Semantic Router more useful for agent-serving stacks where a logical model name hides a model portfolio.

For users, the experience can remain simple: call a model such as auto, send a stable session id, and let the router pick the physical model. For operators, the behavior becomes more controllable: configure when continuity is required, when idle sessions can reset, how much prefix locality matters, and how routing decisions are traced.

This is especially useful when:

• candidate models have different cost, latency, and capability profiles;
• agents use tools across multiple turns;
• clients depend on provider-managed continuation state;
• long sessions build valuable prefix-cache locality;
• operators need to inspect which physical model served each turn behind a logical route.

It also creates a clean infrastructure boundary. Semantic Router owns policy-level model selection. Envoy, Kubernetes, and serving backends still own endpoint membership, health checks, and load balancing. That separation keeps SAAR focused on what it can safely decide: model continuity, model switching, and traceability at the session level.

The Larger Direction​

This milestone continues the same arc that started with Signal-Decision routing.

Iris made routing decisions composable. Athena moved Semantic Router toward a strategic system brain for mixture-of-models and agentic deployments. Multimodal hardening broadened the evidence surface from text prompts to request-level signals. Session-aware agentic routing broadens the time horizon: the router now reasons not only about a request, but about where that request sits inside a long-running interaction.

That direction matters for modern serving stacks. Agent systems increasingly want one logical model interface over many physical options. They want cheaper models for easy steps, stronger models for hard steps, continuity through tool loops, cache-aware handling of long contexts, and enough observability to trust the system in production.

SAAR is a step toward that operating model. It does not make the router an agent. It makes the router aware of the minimum session facts required to serve agents well.

The core idea is simple: a router behind auto should know when a model switch is allowed, when it is forbidden, and what the switch costs for a warm long-running session.

Join Us​

Looking for collaborations! SAAR is the next step in making Semantic Router useful for long-horizon agents, and there is a lot of open work ahead.

We are looking for contributors who want to help with:

• session-aware routing policies for real agent traffic;
• multi-turn and tool-loop evaluation suites;
• AMD ROCm serving validation and performance experiments;
• router observability, replay traces, and production debugging workflows;
• Envoy, Kubernetes, and gateway integrations that keep routing policy separate from endpoint load balancing.

If you are building agent systems, running model portfolios on AMD GPUs, or researching continuity-aware model selection, we would like to collaborate.

Resources:

• GitHub: vllm-project/semantic-router
• Documentation: vllm-semantic-router.com
• Community: join the #semantic-router channel on vLLM Slack

That idea started with text. Iris introduced the Signal-Decision architecture, moving VSR beyond a fixed domain classifier and into a richer system where intent, keywords, embeddings, safety, PII, semantic cache, and plugins can all participate in routing. Athena pushed the same idea further: Semantic Router is not only a fast classifier in front of vLLM, but a system-level intelligence layer for mixture-of-models and agentic deployments.

The next boundary is multimodal routing. Once an image, screenshot, scan, or document page enters the request, the router is no longer reasoning over a prompt alone. It is reasoning over request evidence. The image may be the part that makes the request clinical, regulated, security-sensitive, out of domain, or worth routing to a stronger vision-language model. A router that only sees text is routing a partial request.

This post is about crossing that boundary. The important step is not simply adding an image encoder. The important step is turning visual evidence into a trustworthy VSR signal that can be composed with text signals inside the same decision fabric.

The hardening story below explains why that distinction matters. A deployed multimodal path around multi-modal-embed-small looked confidently wrong. The first explanation seemed obvious: maybe the compact vision encoder was not strong enough. The actual issue was more useful to find and more important for production systems: the Rust/Candle path used by VSR did not match the PyTorch reference path for the same model.

Multimodal routing is not image classification​

Text-only routing already handles more than topic matching. In the Signal-Decision model, signals are independent observations, decisions compose those observations with priority and boolean logic, and plugins or model references define what should happen next. That separation is what lets VSR express policies like "security-sensitive code review gets a stronger reasoning model and jailbreak checks" instead of "computer science goes to the coding model."

Multimodal routing keeps that same shape, but changes the unit of analysis from a text prompt to a full request. The text can be generic while the image carries the decisive evidence:

The innovation is not that VSR can compute an image embedding. The innovation is that the image embedding becomes a typed signal in the same fabric as text intent, PII, jailbreak, domain, semantic similarity, plugins, and model selection. In other words, multimodal support turns VSR from prompt-level routing into request-level policy.

That is also why signal correctness becomes a control-plane requirement. If a text signal is wrong, a policy can route to the wrong model or skip the wrong plugin. If a vision signal is anti-correlated, the problem is worse: the router can become confidently wrong while still leaving a clean, repeatable audit trail for the wrong decision.

Reference parity is therefore not just model-quality hygiene. For VSR, it is a control-plane invariant. The deployed signal path must mean the same thing as the reference model path, or the decision layer is composing the wrong evidence.

When the vision signal was confidently wrong​

The first symptom was not a small accuracy drop. On an 11-image probe across three verticals and 21 candidate labels, the deployed multi-modal-embed-small (mmes) path ranked the wrong vertical highest on 9 of 11 images. Medical X-rays scored closer to semiconductor candidates than to medical candidates. Identifier documents did not reliably land near identifier anchors.

That is an 82% inversion rate. The signal was anti-correlated, not merely noisy.

For a router, this failure mode matters more than a benchmark score. A classifier that is weak usually produces uncertainty. A classifier that is inverted produces confidence in the wrong direction. In a multimodal policy layer, that can be worse than having no image signal at all.

The production surface that exposed the issue was the image-modality routing work around multi-modal-embed-small, including the E2E routing profile introduced in vllm-project/semantic-router PR #1881. Once real images flowed through the Candle binding path, the gap became visible.

The tempting explanation: upgrade the encoder​

The first hypothesis was natural: perhaps the compact encoder was not strong enough for the routing task. Around the same time, the team was already exploring the SigLIP2 family and the larger multi-modal-embed-large (mmEL) direction. That made an encoder upgrade feel like the obvious fix.

We tested that hypothesis directly:

• SigLIP2-base scored 10/10 on the same 21-candidate probe.
• SigLIP-base through Hugging Face Transformers also scored 10/10.
• mmEL, whose vision tower is based on SigLIP2, scored 10/10.
• The mmes model card loaded directly through the PyTorch reference path also scored 10/10.

That result changed the shape of the investigation. The encoder family was not the root problem. Even the supposedly failing mmes model behaved correctly when loaded through the reference path.

There was still useful learning from the encoder chase. The larger SigLIP2-so400m variant showed stronger out-of-distribution rejection in this probe, suppressing an accidentally included car-engine image more aggressively than smaller variants. That may matter for future defensive routing when memory headroom allows a larger vision tower. But it was not the bug behind the inverted production signal.

The reference check that changed the investigation​

The decisive test was simple: run the same mmes model on the same passport fixture through two paths and compare the embedding behavior.

The PyTorch reference path returned cosine 0.7204 against the relevant passport anchor. The deployed Candle-binding path returned 0.1576 on the same image and conceptual pipeline. That is a 5-8x magnitude gap on the same model and fixture.

At that point, the investigation stopped being a model-selection question. The useful question became: where does the production path diverge from the reference path?

The lesson is straightforward: for multimodal routing, reference comparison should be the first diagnostic, not the last. When a production embedding path behaves strangely, compare it against the model card's reference loader before assuming the model itself is too weak.

This is especially important in VSR because the embedding is not only a retrieval primitive. It can become policy evidence. If that evidence has the opposite orientation from the reference model, every downstream layer can be logically correct and operationally wrong.

What was actually broken​

The drift came from implementation details in the Candle path, not from the model weights. Three fixes isolate the problem into concrete layers.

First, the pooling head was wrong. SigLIPVisionEncoder::forward in candle-binding/src/model_architectures/embedding/multimodal_embedding.rs was effectively doing BERT-style mean + Linear + tanh pooling, while SigLIP uses an attentional probe pooling head. PR #1927 mirrors the SigLIP multi-head attention pooling behavior in Candle binding.

Second, the image normalization path was incomplete. The Go image loader produced CHW float32 pixels in [0, 1], while SigLIP expects per-channel normalization equivalent to (x - 0.5) / 0.5. PR #1928 applies that normalization in the Rust encoder path.

Third, preprocessing still carried residual drift after the pooling and normalization fixes. The old Go-side resize path used a 4-tap bilinear implementation. The PyTorch reference path uses PIL-style image preprocessing through SiglipProcessor. PR #1943 moves image decode, resize, and CHW float32 conversion into Rust using the image crate with Catmull-Rom filtering to approximate the PIL bicubic + antialias behavior.

This is the class of bug that is easy to miss in a cross-language serving stack. The Go layer, Rust FFI layer, Candle model implementation, and PyTorch reference can all appear individually reasonable while still producing a route-breaking mismatch end to end.

Validation status​

The numbers below are measurements from the PR branch stack for #1927, #1928, and #1943. They are included as the validation trail for the proposed hardening path. Until all three PRs merge, these numbers should be read as branch-stack validation rather than released production behavior.

A three-vector isolation experiment on the canonical passport fixture (inrule_identifier_passport.jpg) separates model-forward drift from preprocessing drift:

The first row shows that the Rust model-forward path can match the PyTorch reference at fp32-level noise. The remaining drift after the first two fixes lived in preprocessing, which is why moving preprocessing across the FFI boundary matters.

Across a 20-image corpus covering identifier, ambient, code, adversarial, and out-of-distribution examples, the branch-stack measurements are:

• Cosine: min 0.999557, mean 0.999919, max 0.999978
• 20 / 20 images at cosine >= 0.999 vs PyTorch reference
• Pre-fix preprocessing cosine on the canonical fixture was 0.990145