How vLLM Semantic Router Trains Embedding Models and Publishes Them to Hugging Face

2026年7月25日 · 阅读需 6 分钟

Aayush Saini

SDE, Data and AI @ Red Hat

Anup Sharma

AI & Distributed System @ Nutanix

Training embeddings for semantic routing: domain data fine-tuned with LoRA, published to Hugging Face, and consumed by vLLM Semantic Router

vLLM Semantic Router does not only forward prompts to one LLM. It classifies and routes requests using signals — keywords, classifiers, and embeddings.

Embeddings turn text into vectors. The router then:

matches a query to category or anchor prompts (routing signals), or
decides whether a new query is similar enough to reuse a semantic cache hit.

A general-purpose embedder (for example MiniLM) is good on average, but weak on specialized language (medical, law, code). So the project fine-tunes embeddings for those jobs, then publishes them on Hugging Face for everyone to download.