ModelSaar

Routing: requirements, eligibility, ranking

Rendered from docs/routing.md at commit b9d69f7299ed. The repository file is authoritative.

A. Requirements

routing/requirements.py reads the whole request structurally (never the model's output, never instructions inside prompts):

Length is never used to infer difficulty; a short "Is 97 prime?" is math, a 300-sentence "summarize" is summarization.

B. Hard filters (routing/eligibility.py)

Per deployment, each failed check adds a reason and a remedy. Checks: enabled, healthy, role, locality policy (security.locality_mode), alias constraints (require_tags, require_vision, explicit deployments), capabilities (tools, tool_choice_required, parallel_tools, json_schema, json_object, streaming, vision, audio_in, continuation), remote image URLs, effective context (input + output [+ reasoning] <= context_length), max_output_tokens, request fields the backend path cannot honour, and admission feasibility (non-reserving).

Capability values: supported passes; unsupported fails; unknown fails (conservative) with the remedy "run modelsaar probe <id> or declare it under deployments[]".

Explicit deployment selection goes through the same checks.

C. Classification

Deterministic signals handle clear cases at ~0.3–0.6 ms overhead. The optional local classifier (routing.classifier) runs only when enabled and (by default) only when the deterministic result is ambiguous. Contract:

A text-only classifier never claims to inspect image contents; it only sees has_images.

D. Ranking (routing/scoring.py)

For each eligible deployment:

component source label
quality measured success for the task (eval/probe, keyed by fingerprint) → else catalogue/operator prior → else uncalibrated prior (0.5 + small size heuristic, capped 0.75) measured:<n>, prior:<provenance>, uncalibrated[:size-heuristic]
uncertainty 1/sqrt(n) for measured; 0.25 prior; 0.35 uncalibrated
latency_cost measured p50 for the task → else parameter-size heuristic, normalised among candidates latency:measured / size-heuristic
residency_cost 0 if loaded, else normalised load-time estimate (measured or memory heuristic)
overhead_cost normalised memory estimate
affinity_adjust ± routing.switching_penalty for the session's current deployment (hysteresis)

score = w_q·quality − w_lat·latency_cost − w_res·residency_cost − w_over·overhead_cost + affinity

Profiles (routing.profiles): quality (1.0, 0.15, 0.10, 0.05), balanced (1.0, 0.45, 0.30, 0.10), fast (0.6, 1.0, 0.6, 0.2). When the quality of two candidates is within the evidence tolerance (uncertainty/2 for measured, 0.03 for priors) the lower-overhead one wins — which means that with no measurements, auto prefers the cheaper of effectively-tied models. Run modelsaar eval --mode live --save-measurements to replace priors with measurements on your hardware.

For ambiguous requests quality is blended across the candidate task labels weighted by confidence.

E. Affinity and continuity

F. Decision record

{
  "request_id": "req_…", "config_hash": "…", "registry_version": 3,
  "requested_model": "auto", "alias": "auto", "profile": "quality",
  "requirements": {"primary_task": "coding", "tasks": {"coding": 0.7}, "needs_tools": true, "tokens": {...}},
  "exclusions": {"ollama/plain:3b": ["tool calling required; capability tools is unsupported (source_claim)"]},
  "candidates": [{"deployment": "ollama/big:27b", "quality": 0.72, "evidence": "prior:catalogue …", "score": 0.61, "notes": [...]}],
  "chosen": "ollama/big:27b", "evidence": "prior:catalogue",
  "reasons": ["task=coding (clear); required: tool support; context need ~1600 tokens",
              "1 deployment(s) excluded by hard filters; 2 eligible",
              "ollama/big:27b ranked first on profile quality: quality 0.72 (prior:catalogue), latency cost 1.00, not resident"],
  "overhead_ms": 0.41, "outcome": "ok", "failover": []
}

Inspect with modelsaar explain (offline dry run), modelsaar explain --live (running gateway), modelsaar decisions [request_id], GET /admin/decisions, or /admin/ui. Responses carry X-ModelSaar-Request-Id, X-ModelSaar-Deployment, X-ModelSaar-Profile; the model field keeps the requested alias by default (server.response_model_field: deployment to change). Routing commentary is never injected into assistant content.

Embeddings

/v1/embeddings routes an alias defined under embeddings: to the contract's deployments only. Members must be one verified identity: the Ollama digest, or backends[].models[].identity for generic endpoints. The alias's identity is established before any request is dispatched: embeddings[].identity when declared, else the identity recorded for the alias, else the verified identity of the first member in contract order whose identity is known — recorded synchronously at that moment, so two concurrent first requests see the same anchor and cannot bind two embedding spaces (nothing learned from a response ever becomes the anchor). A member with an unknown identity is never used as a replica (a model-name match is not an identity); a member whose identity differs is excluded with a re-indexing remedy, whatever its dimensions — never a silent switch to a different space. When members declare several identities the decision says which one was anchored and asks for an explicit embeddings[].identity. After you change the model deliberately, re-index and set embeddings[].identity to the new value. Request-level dimensions that differ from the alias and encoding_format other than float are rejected (400 unsupported_contract_option). Every response is validated (vector count, indices, dimensions, finite values, unit norm when normalized: true); a violation is 502 embedding_contract_violation, never a 200. query_prefix/document_prefix are not applied by the gateway (inputs are forwarded verbatim) and are rejected when non-empty. A concrete embedding deployment id can also be requested directly.

Memory budgets

A resource_pools[].memory_gb is a hard admission constraint: resident weights (deployments the runtime reports loaded), every distinct deployment currently reserved (incl. an uncertain hold after a timeout/disconnect and the classifier), every deployment assumed resident since its last completed dispatch, and the candidate, plus memory_margin_gb, must fit. A request slot and weight residency are different events: when a response ends the slot is released but the weights stay loaded, so the pool keeps counting them until the runtime's residency observation (Ollama /api/ps on the periodic refresh or /admin/refresh) reconciles the registry. Runtimes that cannot report residency keep the assumption indefinitely unless assumed_residency_ttl_s is set — the gateway cannot see when such a runtime evicts a model, so the conservative choice is the default. A failed residency query (/api/ps errors while the runtime is otherwise healthy) is not an observation: known and assumed occupancy is kept until the query succeeds again. A reservation released because the connection was refused before the request was sent (connect failure) does not assume residency — nothing reached the runtime — so failover to another deployment is not blocked by phantom weights; any outcome where loading or computation may have started keeps the conservative assumption. The estimates are weights-only (memory_gb_estimate with its provenance: Ollama size × 1.15, operator declaration or override); KV cache, workspace and runtime overhead are covered only by the margin, so this is a budget, not an OOM guarantee, and nothing here measures physical memory. Unified memory is counted once per pool; a deployment is counted once however many requests it serves. Unknown estimates follow unknown_memory_policy: allow counts them as 0 and labels the decision; reject refuses admission whenever any counted member — the candidate, a resident or a reservation — has an unknown estimate, because one unknown model makes the budget unverifiable. The check runs atomically with the pool slot; /admin/status shows memory_reserved_gb, the holders and memory_assumed_resident.

Continuity and failover

Pre-commit failover (one attempt, same candidate list) is refused when the request is bound to the deployment that started the exchange: a tool-result turn whose tool-call ids were issued by that deployment, a conversation carrying reasoning_content, or a session pinned by affinity. The decision records failover_allowed and failover_note; the client gets the backend's error and decides. A timeout — before the first byte, while streaming, on an embedding request or inside a capability probe — converts the reservation into an uncertain hold (admission.uncertain_hold_s) because the backend may still be generating; a probe run stops at the first uncertain timeout and reports the probes it skipped. Every upstream call has a deadline: the tighter of server.request_timeout_s and the backend's timeout_s (never "no timeout"). Circuit-breaker success and session pinning are recorded only after a validated completion: a parsed choices[0].message, or a stream that contained at least one completion chunk, no malformed data frame, and a finish reason or [DONE]. SSE comments and keepalives are ignored, usage-only frames are accepted; a data frame that is not a completion structure is not relayed, invalidates the stream (stream_invalid_after_commit, circuit failure, no pin) and is reported in the error frame. The upstream terminal marker is held back: the client receives exactly one data: [DONE], emitted by the gateway after validation, so an error frame always precedes it. An HTTP 200 that is not a completion is 502 upstream_bad_response.