3 篇博文 含有标签「performance」

Long-horizon LLM agents create a routing problem that single-turn prompt routers were not designed to solve. A router still needs to know which model is best for the current request, but it also needs to know when switching models would break the session.

This post introduces Session-Aware Agentic Routing (SAAR), a session-aware model selection policy in vLLM Semantic Router. SAAR keeps semantic routing, but adds router-owned session memory, hard locks around tool loops and non-portable provider state, safe reset boundaries, prefix-cache-aware switch pricing, and replayable traces.

Across 21,600 deterministic turns, SAAR cuts model switches by 79.29%, eliminates 3,836 unsafe switches, and reduces estimated physical-model cost by 78.71%. Across 2,896 live AMD ROCm requests, it preserves session continuity with 0 observed violations.

Figure 1: Long-horizon agents need routing decisions that understand the session trajectory, not only the latest prompt.

From Prompt Routing To Session Routing​

vLLM Semantic Router started from a simple systems observation: not every request should take the same path through an inference stack. A short factual question, a security-sensitive prompt, a multimodal request, a hard reasoning task, and a domain-specific query may all deserve different treatment.

The first generation of that idea was prompt routing. The router extracted signals from the current request, matched a routing decision, and selected an appropriate path. Iris made those signals composable. Athena made the router more strategic by expanding model selection, memory, replay, long-context signals, multimodal primitives, and AMD ROCm deployment paths.

Agents change the unit of routing again.

A coding or research agent is not one prompt. It is a session. It plans, calls tools, receives tool outputs, edits files, runs tests, recovers from errors, pauses, resumes, and often sends very short follow-up messages such as "continue", "fix it", "run that again", or "use the previous result." Those turns are meaningful only because of the trajectory that came before them.

That is why this milestone matters for Semantic Router. The router is no longer answering only:

Which model should handle this request?

For agent traffic, the router also has to answer:

Is it safe to switch models inside this session right now?

That second question is what SAAR is designed to handle.

Why Single-Turn Routing Breaks Down For Agents​

Single-turn routing can be locally correct and still be wrong for the session.

Consider a typical tool-using agent loop:

The latest message alone does not contain enough information. A prompt router may decide that the tool result looks cheap and send it to a smaller model. It may see a generic "continue" and re-run the normal selector. It may miss that provider-managed continuation state belongs to one physical backend. It may discard a warm prefix cache for a frontier model because the current message is short.

Each of those mistakes has a different failure mode:

• A tool result can go to a model that did not make the tool call.
• A non-portable continuation id can be sent to the wrong physical backend.
• A long, warm session can lose prefix locality and become unnecessarily expensive.
• A logical model such as auto can become hard to debug because users no longer know which physical model actually served the turn.

The important point is not that agents should never switch models. They should. A good router should still move from a cheap model to a stronger model when the task becomes harder, and it should move back when the session reaches a safe boundary. The problem is that the router needs session context to know which moments are safe.

The SAAR Design​

SAAR keeps the existing Semantic Router decision pipeline. Signals are still extracted from the request, decisions are still matched, and model-selection algorithms still rank candidate models inside a matched decision.

SAAR adds a session-control layer around that result.

Figure 2: SAAR combines router memory, hard locks, reset boundaries, switch economics, and replayable traces before selecting a physical model.

There are five pieces:

This is a model-selection policy, not an endpoint load balancer. Semantic Router can choose a model or cluster through the gateway contract. Endpoint membership, health checks, and load balancing inside a cluster remain infrastructure responsibilities.

The Most Important Rule: Sometimes The Router Must Not Switch​

The safest model switch is not always the one with the best score on the latest prompt. For agent traffic, some turns are continuity-constrained.

Figure 3: Tool loops and provider-managed continuation state are hard continuity constraints; idle and decision-drift boundaries permit safe reselection.

SAAR treats two cases as hard locks:

• Tool-loop continuity. If a physical model asked for a tool call, the tool result should return to that same physical model. The follow-up observation is not a fresh prompt; it is part of a local execution loop.
• Provider-managed state. If the request carries non-portable continuation state, such as a response identifier that belongs to one backend, SAAR holds the previous physical model instead of silently moving the state elsewhere.

These rules are intentionally stronger than cost rules. If a switch is unsafe, the router should not "buy" its way out with a cheaper model.

SAAR also defines the opposite boundary: when the router may switch again. Idle timeout and decision drift reopen the selection. If an agent pauses long enough, the value of continuity decays. If the matched decision changes because the user moved from code editing to synthesis or from retrieval to debugging, the old model choice should not stick forever.

This distinction is the heart of session-aware agentic routing:

Router Memory Is Not User Memory​

The phrase "router memory" can be misleading, so the boundary is important.

SAAR memory is not conversation memory, retrieval memory, or user profile memory. It does not summarize the conversation and it does not try to remember facts for the model. Its job is narrower: keep enough routing state to make the next model-selection decision safe and explainable.

For each session, the router tracks facts such as:

• the last physical model selected behind the logical model;
• the last matched routing decision;
• whether the session is in a normal, tool-loop, provider-state, idle-reset, or drift-reset phase;
• how many recent switches happened;
• the latest context length and cache evidence;
• a replay id that links the response back to the router's decision trace.

That scope keeps the system operationally useful without turning the router into a second agent memory layer. Application memory should remain in the application. Retrieval memory should remain in the retrieval stack. SAAR memory exists only to make routing across turns coherent.

Prefix Cache Makes Model Switching Asymmetric​

For long agent sessions, model switching is not just a quality decision. It is also an input-side systems decision.

Figure 4: The same switch has a different cost depending on model tier, session length, and physical prefix reuse.

A short retry on a cheap model and a 40-turn warm session on a frontier model should not be treated the same way. The latter has accumulated a valuable prefix. Switching away from it may require the next physical model to pay a much larger input cost even if the visible user message is short.

SAAR therefore prices a cached-input checkout delta: the gap between normal prompt input price and cached-input price for the physical model under consideration. The longer and more expensive the session, the stricter the policy becomes about discarding prefix locality.

This also clarifies cached-token accounting for a routed logical model. If the user calls auto, the router may map that logical name to different physical models over time. A cache hit reported by one backend is physical evidence for that backend. It is not automatically transferable to another backend. SAAR keeps backend-reported cached tokens separate from router-estimated reuse, and it does not rewrite upstream usage fields.

That separation is useful operationally. Operators can still inspect physical cache behavior while the router uses its own memory to decide whether switching is worth the checkout cost.

How A Request Moves Through SAAR​

The serving path stays familiar. Clients send requests to the OpenAI-compatible gateway, usually with a logical model name such as auto. To enable session-aware routing, they also send a stable session identifier such as x-session-id.

SAAR then handles each turn in this order:

1. Read the current request, session id, tool-call context, provider-state markers, and candidate model set.
2. Run the normal Semantic Router signal and decision pipeline.
3. Produce a base model-selection result from the configured method, such as hybrid scoring.
4. Load the previous session routing state from router memory.
5. Apply hard locks for tool loops and provider-managed state.
6. Check idle timeout and decision drift boundaries.
7. Adjust switch scores using prefix-cache checkout cost and switch history.
8. Select the physical model and emit diagnostics.
9. Update router memory and write a replay trace.

The configuration lives inside a routing decision's model-selection algorithm:

routing:
  decisions:
    - name: agentic_routing
      modelRefs:
        - model: qwen3-8b
        - model: qwen3-32b
      algorithm:
        type: session_aware
        session_aware:
          base_method: hybrid
          idle_timeout_seconds: 300
          tool_loop_hard_lock: true
          context_portability_hard_lock: true
          decision_drift_reset: true
          prefix_cache_weight: 0.20
          switch_history_weight: 0.04

The values are intentionally policy knobs, not one-size-fits-all constants. A customer-service assistant with short sessions may use a more permissive idle boundary. A coding agent with long tool loops and expensive context may use stricter continuity and prefix-cache settings.

Observability Is Part Of The Feature​

Model selection behind auto is only useful if operators can explain it.

Figure 5: SAAR turns hidden physical routing choices behind a logical model into inspectable traces and response headers.

SAAR emits diagnostics such as selected model, selected decision, replay id, session phase, selected confidence, and context-token count. The replay id joins a served response back to the router trace that explains the decision.

A useful trace answers questions like:

• What model would the base selector have chosen?
• Did the router hold the previous model because of a tool-loop lock?
• Did provider-managed state make switching unsafe?
• Did the session cross an idle or drift boundary?
• How did prefix-cache evidence change the adjusted candidate scores?
• Was the final decision a stay, a switch, or a locked stay?

This makes session-aware routing operable. Without replay, a router behind a logical model becomes hard to debug. With replay, an operator can audit why the router preserved continuity or decided that a switch was safe.

How We Evaluate It​

The evaluation is designed around one question: does the policy make routing more agent-friendly without hiding correctness problems?

We use three layers of evidence.

First, a deterministic policy matrix tests the control logic across many synthetic sessions. This isolates the routing policy from serving noise and lets us stress tool loops, provider state, idle boundaries, drift boundaries, model tiers, and switch history.

Second, live OpenAI-compatible serving runs exercise the same invariants through the router and backend serving path on AMD ROCm. This checks that headers, session ids, diagnostics, and failure handling survive real request flow.

Third, deterministic agent-task traces add task structure. Instead of only counting switches, these traces include simulated tool observations and exact final-answer scoring.

The goal is not to make every plot say "fewer switches." Sticky sessions can do that. The goal is to show that SAAR removes unsafe switches, keeps useful movement, respects expensive prefix locality, and remains observable in live serving.

Result 1: SAAR Moves The Unit Of Control From Turn To Session​

The deterministic policy matrix covers balanced, tool-heavy, frontier-heavy, idle-heavy, provider-state-heavy, and drift-heavy sessions. Each workload runs five seeds, 40 sessions per seed, and 18 turns per session, for 21,600 total turns.

Figure 6: Headline policy result across 21,600 deterministic turns.

The headline result is that SAAR reduces model churn while preserving the ability to move:

Single-turn routing switches often and creates unsafe movement. Sticky sessions nearly eliminate movement, but they also give up too much quality because they refuse to reselect after the task changes. Full SAAR sits in the middle for the right reason: it removes unsafe movement while still letting idle and drift boundaries reopen the decision.

This is the shift from turn-level control to session-level control. The router is no longer treating every message as a fresh independent event.

Result 2: Hard Locks Remove The Correctness Failures​

The second result isolates the most important invariant: when switching is unsafe, SAAR should not switch.

Figure 7: Hard locks remove unsafe switching during tool loops and non-portable provider state.

Tool-loop switch violations fall from 3,404 to 0. Provider-state switch violations fall from 432 to 0.

These are not minor tuning wins. They are correctness boundaries. A tool result is not an ordinary prompt. A non-portable continuation id is not an ordinary text field. If a router ignores those facts, it can break the interaction while still appearing to make a reasonable semantic choice on the latest message.

SAAR fixes that by making continuity constraints explicit in the policy.

Result 3: SAAR Is Not Sticky Sessions With A New Name​

The obvious baseline is sticky routing: pick the first model for a session and hold it.

Sticky routing is attractive because it is easy to reason about. It also solves many unsafe switch cases by avoiding switching entirely. But that simplicity becomes a product problem for agents. Long sessions drift. Users change tasks. A cheap initial model may no longer be appropriate. A strong model may no longer be necessary.

Figure 8: SAAR is not just sticky sessions; it balances continuity with movement.

The ablation shows why SAAR needs multiple mechanisms:

Locks provide correctness. Reset boundaries provide liveness. Prefix-cache checkout pricing provides economic discipline. Removing any one of them changes the behavior in a way that is visible in the metrics.

Result 4: The Invariants Hold In Live AMD ROCm Serving​

Policy simulation is useful, but a router has to work through real request flow. The live serving runs use OpenAI-compatible traffic through the router and AMD ROCm backend paths, with matched schedules for routed and direct-backend runs.

Figure 9: Live ROCm runs preserve continuity under long sessions and injected backend failures.

Across long-session runs, the router completes 2,896 live requests with 0 observed continuity violations.

The idle workload includes real wall-clock sleeps, so its p95 overhead should be interpreted separately from hot-path routing overhead. The important result is continuity: the live path preserves the hard-lock and reset-boundary behavior tested in the deterministic matrix.

Result 5: Sessions Recover After Backend Faults​

Long-horizon agents need session-level recovery, not just per-request success. A backend can return HTTP 503 for one request, but the session should continue later without losing the routing invariants that keep the interaction coherent.

Across the one-shot disruption matrix, 32/32 affected sessions recovered later. Across the repeated-failure matrix, 24/24 affected sessions recovered after 168 injected HTTP 503 responses.

This matters because agent sessions are longer than ordinary chat turns. A transient backend fault should not make the router forget that a tool loop is active, that provider state is non-portable, or that the session has a replayable history.

Result 6: Task Traces Exercise The Agent Loop​

Continuity counters are necessary, but they are not enough. We also run deterministic multi-turn task traces with simulated tool observations and exact final-answer scoring. There is no judge model in the loop: the final answer either contains the required labels or it does not.

In the AMD serving task run, 18/18 exact-scored task instances complete, replay headers are present on 96/96 routed turns, and no continuity violation is observed.

This is still smaller than a broad real coding-agent benchmark, but it is a stronger signal than policy counters alone because it exercises a task loop with tool observations and final-answer checks.

What This Changes For vLLM Users​

Session-aware routing makes Semantic Router more useful for agent-serving stacks where a logical model name hides a model portfolio.

For users, the experience can remain simple: call a model such as auto, send a stable session id, and let the router pick the physical model. For operators, the behavior becomes more controllable: configure when continuity is required, when idle sessions can reset, how much prefix locality matters, and how routing decisions are traced.

This is especially useful when:

• candidate models have different cost, latency, and capability profiles;
• agents use tools across multiple turns;
• clients depend on provider-managed continuation state;
• long sessions build valuable prefix-cache locality;
• operators need to inspect which physical model served each turn behind a logical route.

It also creates a clean infrastructure boundary. Semantic Router owns policy-level model selection. Envoy, Kubernetes, and serving backends still own endpoint membership, health checks, and load balancing. That separation keeps SAAR focused on what it can safely decide: model continuity, model switching, and traceability at the session level.

The Larger Direction​

This milestone continues the same arc that started with Signal-Decision routing.

Iris made routing decisions composable. Athena moved Semantic Router toward a strategic system brain for mixture-of-models and agentic deployments. Multimodal hardening broadened the evidence surface from text prompts to request-level signals. Session-aware agentic routing broadens the time horizon: the router now reasons not only about a request, but about where that request sits inside a long-running interaction.

That direction matters for modern serving stacks. Agent systems increasingly want one logical model interface over many physical options. They want cheaper models for easy steps, stronger models for hard steps, continuity through tool loops, cache-aware handling of long contexts, and enough observability to trust the system in production.

SAAR is a step toward that operating model. It does not make the router an agent. It makes the router aware of the minimum session facts required to serve agents well.

The core idea is simple: a router behind auto should know when a model switch is allowed, when it is forbidden, and what the switch costs for a warm long-running session.

Join Us​

Looking for collaborations! SAAR is the next step in making Semantic Router useful for long-horizon agents, and there is a lot of open work ahead.

We are looking for contributors who want to help with:

• session-aware routing policies for real agent traffic;
• multi-turn and tool-loop evaluation suites;
• AMD ROCm serving validation and performance experiments;
• router observability, replay traces, and production debugging workflows;
• Envoy, Kubernetes, and gateway integrations that keep routing policy separate from endpoint load balancing.

If you are building agent systems, running model portfolios on AMD GPUs, or researching continuity-aware model selection, we would like to collaborate.

Resources:

• GitHub: vllm-project/semantic-router
• Documentation: vllm-semantic-router.com
• Community: join the #semantic-router channel on vLLM Slack

Most routing systems start with a prompt and choose a model endpoint. vLLM Semantic Router (VSR) makes a different bet: before a request reaches the serving model, the system should extract signals, compose those signals into decisions, and make the chosen path observable, auditable, and programmable.

That idea started with text. Iris introduced the Signal-Decision architecture, moving VSR beyond a fixed domain classifier and into a richer system where intent, keywords, embeddings, safety, PII, semantic cache, and plugins can all participate in routing. Athena pushed the same idea further: Semantic Router is not only a fast classifier in front of vLLM, but a system-level intelligence layer for mixture-of-models and agentic deployments.

The next boundary is multimodal routing. Once an image, screenshot, scan, or document page enters the request, the router is no longer reasoning over a prompt alone. It is reasoning over request evidence. The image may be the part that makes the request clinical, regulated, security-sensitive, out of domain, or worth routing to a stronger vision-language model. A router that only sees text is routing a partial request.

This post is about crossing that boundary. The important step is not simply adding an image encoder. The important step is turning visual evidence into a trustworthy VSR signal that can be composed with text signals inside the same decision fabric.

The hardening story below explains why that distinction matters. A deployed multimodal path around multi-modal-embed-small looked confidently wrong. The first explanation seemed obvious: maybe the compact vision encoder was not strong enough. The actual issue was more useful to find and more important for production systems: the Rust/Candle path used by VSR did not match the PyTorch reference path for the same model.

Multimodal routing is not image classification​

Text-only routing already handles more than topic matching. In the Signal-Decision model, signals are independent observations, decisions compose those observations with priority and boolean logic, and plugins or model references define what should happen next. That separation is what lets VSR express policies like "security-sensitive code review gets a stronger reasoning model and jailbreak checks" instead of "computer science goes to the coding model."

Multimodal routing keeps that same shape, but changes the unit of analysis from a text prompt to a full request. The text can be generic while the image carries the decisive evidence:

The innovation is not that VSR can compute an image embedding. The innovation is that the image embedding becomes a typed signal in the same fabric as text intent, PII, jailbreak, domain, semantic similarity, plugins, and model selection. In other words, multimodal support turns VSR from prompt-level routing into request-level policy.

That is also why signal correctness becomes a control-plane requirement. If a text signal is wrong, a policy can route to the wrong model or skip the wrong plugin. If a vision signal is anti-correlated, the problem is worse: the router can become confidently wrong while still leaving a clean, repeatable audit trail for the wrong decision.

Reference parity is therefore not just model-quality hygiene. For VSR, it is a control-plane invariant. The deployed signal path must mean the same thing as the reference model path, or the decision layer is composing the wrong evidence.

When the vision signal was confidently wrong​

The first symptom was not a small accuracy drop. On an 11-image probe across three verticals and 21 candidate labels, the deployed multi-modal-embed-small (mmes) path ranked the wrong vertical highest on 9 of 11 images. Medical X-rays scored closer to semiconductor candidates than to medical candidates. Identifier documents did not reliably land near identifier anchors.

That is an 82% inversion rate. The signal was anti-correlated, not merely noisy.

For a router, this failure mode matters more than a benchmark score. A classifier that is weak usually produces uncertainty. A classifier that is inverted produces confidence in the wrong direction. In a multimodal policy layer, that can be worse than having no image signal at all.

The production surface that exposed the issue was the image-modality routing work around multi-modal-embed-small, including the E2E routing profile introduced in vllm-project/semantic-router PR #1881. Once real images flowed through the Candle binding path, the gap became visible.

The tempting explanation: upgrade the encoder​

The first hypothesis was natural: perhaps the compact encoder was not strong enough for the routing task. Around the same time, the team was already exploring the SigLIP2 family and the larger multi-modal-embed-large (mmEL) direction. That made an encoder upgrade feel like the obvious fix.

We tested that hypothesis directly:

• SigLIP2-base scored 10/10 on the same 21-candidate probe.
• SigLIP-base through Hugging Face Transformers also scored 10/10.
• mmEL, whose vision tower is based on SigLIP2, scored 10/10.
• The mmes model card loaded directly through the PyTorch reference path also scored 10/10.

That result changed the shape of the investigation. The encoder family was not the root problem. Even the supposedly failing mmes model behaved correctly when loaded through the reference path.

There was still useful learning from the encoder chase. The larger SigLIP2-so400m variant showed stronger out-of-distribution rejection in this probe, suppressing an accidentally included car-engine image more aggressively than smaller variants. That may matter for future defensive routing when memory headroom allows a larger vision tower. But it was not the bug behind the inverted production signal.

The reference check that changed the investigation​

The decisive test was simple: run the same mmes model on the same passport fixture through two paths and compare the embedding behavior.

The PyTorch reference path returned cosine 0.7204 against the relevant passport anchor. The deployed Candle-binding path returned 0.1576 on the same image and conceptual pipeline. That is a 5-8x magnitude gap on the same model and fixture.

At that point, the investigation stopped being a model-selection question. The useful question became: where does the production path diverge from the reference path?

The lesson is straightforward: for multimodal routing, reference comparison should be the first diagnostic, not the last. When a production embedding path behaves strangely, compare it against the model card's reference loader before assuming the model itself is too weak.

This is especially important in VSR because the embedding is not only a retrieval primitive. It can become policy evidence. If that evidence has the opposite orientation from the reference model, every downstream layer can be logically correct and operationally wrong.

What was actually broken​

The drift came from implementation details in the Candle path, not from the model weights. Three fixes isolate the problem into concrete layers.

First, the pooling head was wrong. SigLIPVisionEncoder::forward in candle-binding/src/model_architectures/embedding/multimodal_embedding.rs was effectively doing BERT-style mean + Linear + tanh pooling, while SigLIP uses an attentional probe pooling head. PR #1927 mirrors the SigLIP multi-head attention pooling behavior in Candle binding.

Second, the image normalization path was incomplete. The Go image loader produced CHW float32 pixels in [0, 1], while SigLIP expects per-channel normalization equivalent to (x - 0.5) / 0.5. PR #1928 applies that normalization in the Rust encoder path.

Third, preprocessing still carried residual drift after the pooling and normalization fixes. The old Go-side resize path used a 4-tap bilinear implementation. The PyTorch reference path uses PIL-style image preprocessing through SiglipProcessor. PR #1943 moves image decode, resize, and CHW float32 conversion into Rust using the image crate with Catmull-Rom filtering to approximate the PIL bicubic + antialias behavior.

This is the class of bug that is easy to miss in a cross-language serving stack. The Go layer, Rust FFI layer, Candle model implementation, and PyTorch reference can all appear individually reasonable while still producing a route-breaking mismatch end to end.

Validation status​

The numbers below are measurements from the PR branch stack for #1927, #1928, and #1943. They are included as the validation trail for the proposed hardening path. Until all three PRs merge, these numbers should be read as branch-stack validation rather than released production behavior.

A three-vector isolation experiment on the canonical passport fixture (inrule_identifier_passport.jpg) separates model-forward drift from preprocessing drift:

The first row shows that the Rust model-forward path can match the PyTorch reference at fp32-level noise. The remaining drift after the first two fixes lived in preprocessing, which is why moving preprocessing across the FFI boundary matters.

Across a 20-image corpus covering identifier, ambient, code, adversarial, and out-of-distribution examples, the branch-stack measurements are:

• Cosine: min 0.999557, mean 0.999919, max 0.999978
• 20 / 20 images at cosine >= 0.999 vs PyTorch reference
• Pre-fix preprocessing cosine on the canonical fixture was 0.990145

The important result is not just the final cosine number. It is the isolation method: compare the production path against the reference path, split model-forward drift from preprocessing drift, then make the production path use the same preprocessing semantics in tests and serving.

What this unlocks for VSR​

Once the vision path is trustworthy, VSR can treat images as first-class evidence rather than side-channel metadata. That unlock is larger than "route image requests to an image model." It lets text and image evidence participate in the same Signal-Decision fabric:

This is the natural continuation of the Iris and Athena direction. Iris made routing decisions composable. Athena made the router more strategic by adding a stronger model stack, model selection, memory, replay, and richer signal handling. Multimodal routing extends that same architecture from language-only control to request-level control.

The public demo associated with this work is shrader.dev. Today it demonstrates the text-routing version of the policy pattern: domain relevance checks, privacy-sensitive routing, and blocked outcomes before model invocation. That demo is important because it shows the policy shape before images are added.

The text-routing path also illustrates a performance property that matters for multimodal production. Classifier signals can run concurrently through runSignalDispatchers, so wall-clock latency is bounded by the slowest enabled classifier rather than the sum of all classifiers. In a representative trace, the full classification decision completes in roughly 1.3 seconds on CPU.

The multimodal version of that story is not a separate product path. It is the same policy engine with a larger evidence surface. Image and text signals should be extracted, validated, composed, replayed, and audited through the same routing semantics.

That is why the hardening work matters. If VSR is going to route on visual evidence, the vision signal path has to be boringly reliable. It must match the reference model, survive cross-language serving boundaries, and remain testable as policies become more expressive.

What comes next​

The immediate work is to land and review the hardening PRs, then keep the validation corpus in the loop as multimodal routing evolves. The larger direction is to make reference-driven checks a normal part of VSR's multimodal serving story.

From there, the next steps are architectural:

• expose image-derived signals in the same decision layer as text-derived signals;
• keep multimodal decisions visible in replay, metrics, and debugging tools;
• make model selection aware of both policy fit and modality capability;
• preserve high-fidelity inspection for safety-critical signals such as PII and jailbreak;
• extend the same fabric toward agentic workflows, where tool calls, memory writes, and model invocations are routed through one decision layer.

Text routing was the first control surface. Multimodal routing is the next one. The goal is not to build a one-off visual classifier beside the router, but to make every meaningful part of a request available to the same programmable routing brain.

Getting started:

• Project repository: vllm-project/semantic-router
• Live demo: shrader.dev

Acknowledgments​

Thanks to Huamin Chen for the mmEL pointer that helped break the encoder-upgrade misdiagnosis, the maintainer reviews across #1927, #1928, and #1943, and the invitation to write this up. Thanks also to the broader maintainer team for the multi-modal classifier work this arc plugs into, the multi-modal-embed-small model card, and the Candle-binding integration this all builds on.

Anthropic recently published an insightful blog post on code execution with MCP, highlighting a critical challenge in modern AI systems: as agents connect to more tools, loading all tool definitions upfront becomes increasingly inefficient. Their solution—using code execution to load tools on-demand—demonstrates how established software engineering patterns can dramatically improve agent efficiency.

This resonates deeply with our experience building the vLLM Semantic Router. We've observed the same problem from a different angle: when AI agents have access to hundreds or thousands of tools, how do they know which tools are relevant for a given task?

Our solution: semantic tool selection—using semantic similarity to automatically select the most relevant tools for each user query before the request even reaches the LLM.