How it works

ModelSaar is one Python process that speaks the OpenAI Chat Completions API on one side and talks to your inference backends on the other. In between, every request passes through the same deterministic pipeline, and every outcome is written down.

In plain language

1. Read the request. Does it carry tools, an image, a required output schema, a long history, a half-finished tool call? Roughly how many tokens will it need, including the answer?
2. Cross out what cannot do the job. Each configured model is checked against those needs and against your policy: is it healthy, local (if you require local), able to call tools, able to read images, big enough in context, allowed by the alias, not over capacity? A model with an unknown capability is crossed out for requests that need it — unknown never counts as yes.
3. Rank what is left. Measured task success where you have run the evaluation, otherwise labelled priors, minus costs for latency, loading an unloaded model and switching away from the model a conversation is already on.
4. Send the request unchanged to the winner, stream the answer back, and keep the conversation and its tool calls on that model.
5. Write the decision down — which models were excluded and why, who won and on what evidence, how long the decision took — without the prompt.

Below, the same thing for people evaluating the design. Terms link to the repository documentation, which is authoritative.

1. Requirements

The whole request body is analysed structurally (routing/requirements.py): message roles; text, image and audio parts (typed — unknown part types such as audio_url, video_url or file are rejected with 400 unsupported_content_type before anything is sent anywhere); tools and tool_choice; response_format (JSON object or strict schema); outstanding tool-call ids from the last assistant turn; reasoning_content that implies continuation state; and a conservative token budget — input estimate (chars/3 for Latin text, 1 token per CJK character, plus per-message, per-tool and per-image overheads, +5 %), the requested output allowance, and a reasoning allowance for deployments that can think.

Deterministic multi-label task signals (conversation, coding, agent, math, extraction, summarization, translation, rag, decision, document_vision) give a primary task label. An optional local classifier can refine the label when the deterministic result is ambiguous; it has bounded input, a strict JSON schema, a timeout and a deterministic fallback, and it must be a local deployment unless you opt in twice (routing.classifier.allow_remote and security.locality_mode: allow_remote). It can only change the task label — never policy, endpoints or eligibility.

2. Hard eligibility filters

A failed filter can never be outscored (routing/eligibility.py). In order: role (generative vs embedding), enabled, healthy, locality policy, alias restrictions (required tags, vision, explicit deployment list), capability tri-state for everything the request needs (tools, parallel tools, tool_choice=required, JSON object, strict schema, streaming, image input, audio, continuation), remote image URLs (never fetched), effective context (unknown context ⇒ not eligible), max_output_tokens, request fields the backend path cannot honour (on Ollama: tool_choice=required/named, logit_bias, n, logprobs), and admission feasibility (open circuit, full queue, pool memory budget).

Capabilities are three-state — supported, unsupported, unknown — with provenance: the backend's own metadata (/api/show for Ollama), the adapter path, a probe (modelsaar probe runs short real requests for streaming, tools, parallel tools, JSON object, strict schema, image input and latency), or an operator override declared in the config. Evidence is keyed by a fingerprint of the deployment's identity and serving configuration (backend id, endpoint, model name, digest/identity, runtime and version, template, quantisation, pool); change any of those and the stored evidence no longer applies, while loading or unloading a model does not change it.

Token accounting is a two-pass affair for privacy: estimates first; only deployments that already passed every policy and capability filter may be asked for a tokenizer count (vLLM /tokenize), under one deadline, and the result is labelled tokenize:<backend>+template-estimate because template and image overheads remain estimates.

3. Ranking, profiles and aliases

Each eligible candidate gets a transparent score (routing/scoring.py): quality for the task — measured success from the evaluation harness (with sample count), else a prior from the catalogue or operator, else an uncalibrated prior (0.5 plus a small, capped size heuristic) — minus weighted costs for measured latency, residency/loading and overhead, plus a switching penalty when a session is already on another deployment. Effectively tied candidates resolve to the lower-overhead one. The decision record shows every candidate's breakdown and the evidence level used.

Profiles weight those terms: quality (default), balanced, fast. Aliases are the virtual models an app selects: auto, auto-quality, auto-balanced, auto-fast, auto-code (deployments tagged code), auto-vision (verified image input). An alias is advertised in GET /v1/models only when at least one usable deployment backs it, with a truthful contract (minimum context, tools/vision/schema = supported, partial or unsupported). A concrete deployment id can always be requested directly and still passes the same hard checks.

4. Continuity, capacity and failure

• Sessions. With a session key (X-ModelSaar-Session, X-OpenWebUI-Chat-Id, or prompt_cache_key in the body) a conversation is pinned to the deployment that answered it, scoped per API key. A pinned deployment that no longer qualifies is re-routed, and the record says so.
• Tool round trips. The deployment that issued a tool call's ids receives the tool-result turn. If it cannot, the request fails (affinity_broken) rather than handing a tool history to a model that never saw it.
• Failover happens at most once, only before any byte has reached the client, only on retryable transport errors, and never inside a tool round trip, with reasoning-continuation state, or on a pinned session (failover_allowed: false with the reason). After the response is committed a failing stream is terminated with an error frame — the gateway never switches models mid-answer.
• Admission. Per-backend and per-pool concurrency slots with bounded queues; a client that disconnects mid-stream, or a backend that times out, leaves an uncertain hold on the slot for a bounded time because the backend may still be generating. A pool memory_gb is a hard budget on weights-only estimates plus a margin (it prevents clearly oversized dispatches; it is not an OOM guarantee). Per-backend circuit breakers count only validated completions as success.
• Completion contract. An upstream HTTP 200 that is not a completion envelope (a proxy's HTML page, a JSON error) is a 502 upstream_bad_response, never relayed as success; a stream must end with a finish reason or [DONE] to count.
• Probes under admission. modelsaar probe goes through POST /admin/probe while a gateway runs, so verification never races your traffic; direct probing next to a listening gateway is refused unless you accept it explicitly.

5. Embeddings

/v1/embeddings routes an alias to the deployments of its contract only. Members must be one verified identity (the Ollama digest, or a declared identity for generic servers); a model-name match is not an identity, and a changed identity requires an explicit re-indexing decision. Request options that would change the vector space (dimensions, non-float encoding_format) are rejected; every response's vector count, indices, dimensions, finiteness and (when declared) unit norm are validated, and a violation is a 502 embedding_contract_violation, never a 200.

6. The decision record

Every request — routed or refused — produces a record: request id, config hash, requested alias, profile, inferred task and requirements, exclusions with reasons and remedies, candidate scores and evidence levels, the chosen deployment, affinity and failover state, translation notes (what the backend path changed, e.g. dropping tools for tool_choice=none), outcome and overhead in milliseconds. It contains no prompt text. Reach it through the response headers (X-ModelSaar-Request-Id, X-ModelSaar-Deployment, X-ModelSaar-Profile), GET /admin/decisions/<id>, modelsaar decisions, modelsaar explain (dry run) or the status page at /admin/ui. Records are kept in a local SQLite file for a configurable number of days.

7. Supported endpoints and limits

Reproduce an evaluation with modelsaar eval --mode live --save-measurements (see Benchmarks); read the authoritative documentation in Routing, Architecture and Privacy and security.